Suggestions for improving Etlworks Integrator.

42 votes

Packages and versioning

It would be nice to create a "package" of flows and connections and store it as a certain version. It should then be easy to switch between versions just by toggling ...
Suggested by: Franck van der Sluis (19 Apr, '19) • Upvoted: 12 Aug, '20 • Comments: 3
Done roadmap
10 votes

SSO Authentication

Etlworks handles and processes a lot of sensitive data. It is important that this is adequately shielded and becomes part of our company-wide security umbrella. The ...
Suggested by: Joris Hooi (24 Oct, '19) • Upvoted: 05 May, '22 • Comments: 1
Done roadmap
10 votes

The ability to manage tags: edit, delete

Suggested by: Rupesh Sawant (23 Jun, '20) • Upvoted: 22 Sep, '22 • Comments: 0
Planned roadmap
6 votes

Ability to schedule and execute flows by data integration agents running behind the firewall

Basically, add support for hybrid data integration when flows can be executed on data integration agents running behind the firewall.
Suggested by: Olena Sherbinin (23 Mar, '19) • Upvoted: 27 Dec, '21 • Comments: 1
Done roadmap
6 votes

New fields' mapping editor

Enhance field mapping editor: - add support for nested fields in the destination - add an ability to modify attributes, such as JSON array, XML CDATA, XML ...
Suggested by: Etlworks (06 Dec, '18) • Upvoted: 17 May, '22 • Comments: 0
Planned roadmap
5 votes

Better ways to determine how flows, connections, etc. are interconnected

Right now when you have multiple flows that share connections or other flows, it's difficult to understand how making a change to one will possibly impact other ...
Suggested by: Shanon (07 Jul, '19) • Upvoted: 28 Sep, '19 • Comments: 5
Done roadmap
5 votes

Implement priority queue and throttling for user-defined APIs (listeners)

Scenario 1: upstream system slammed the system with thousands of API calls in a short span of time, eventually freezing and requiring a reboot to stop all jobs ...
Suggested by: NBCUniversal (07 Nov, '20) • Upvoted: 08 Mar, '21 • Comments: 3
Done roadmap
5 votes

Usability enhancement for Flows Builder

1. Ability to edit inner flow from the nested flow 2. Ability to stop API flow, also stop flow from scheduler 3. Add Find Usage for flows (within nested flows)
Suggested by: Maksym (03 Oct, '19) • Upvoted: 08 Jun, '20 • Comments: 0
Planned roadmap
5 votes

Drag and drop process builder for nested flows

Replace existing nested flow editor with drag and drop process builder: - works similarly to the diagram editor - ability to select existing flows from the side-bar ...
Suggested by: Etlworks (06 Dec, '18) • Upvoted: 31 Jan, '22 • Comments: 0
Planned roadmap
5 votes

Provisioning from AWS and Azure Marketplaces

Add ability to provision Etlworks from AWS and Azure Marketplaces with Amazon and Microsoft billing
Suggested by: Etlworks (06 Dec, '18) • Upvoted: 13 Jan • Comments: 0
Planned roadmap
4 votes

AWS Kinesis Integration

AWS Kinesis Integration
Suggested by: Karthik (08 Jun, '20) • Upvoted: 14 Jul, '20 • Comments: 1
Done roadmap
4 votes

Add an apply to all button when editing a transform

Apply the settings for that transform to all the other transforms in the current flow
Suggested by: Bryan (11 Mar, '20) • Upvoted: 23 Apr, '21 • Comments: 0
Planned roadmap
4 votes

Auto-completion in JavaScript code editors

Add auto-completion in JavaScript code editors: - {field names} - functions
Suggested by: Etlworks (06 Dec, '18) • Upvoted: 24 May, '22 • Comments: 0
Planned roadmap
3 votes

Scheduler queue

Introduce queuing in scheduler so you can set how many scheduled tasks can execute simultaneously, and if one tries to run while another is running it will wait till ...
Suggested by: Bryan (18 Mar, '20) • Upvoted: 12 Oct, '22 • Comments: 2
Done roadmap
3 votes

Add support for JMS, ActiveMQ and AWS SQS

Suggested by: Etlworks (03 Oct, '19) • Upvoted: 14 Jul, '21 • Comments: 2
Done roadmap
3 votes

Bulk insert with db to db flows

Having inserts into postgres and sqlserver work like redshift would be a nice improvement. Right now it runs a bunch of insert statements. Instead, pull source data ...
Suggested by: Bryan Meyerovich (10 Oct, '20) • Upvoted: 15 Nov, '20 • Comments: 1
Done roadmap
3 votes

Import entire schemes without specifying individual table

I really liked how aws dms operated, you pointed it at a schema on your source, told it the target schema in your destination, and had a wildcard supporting ...
Suggested by: Bryan (24 Jul, '19) • Upvoted: 28 Sep, '19 • Comments: 1
Done roadmap
3 votes

Allow setting flow parameter in explorer

Made a api connector with a {param1} in the url, want to do this in explorer: select * from content where param1=123
Suggested by: Bryan (31 Aug, '21) • Upvoted: 26 Sep, '21 • Comments: 0
Planned roadmap
3 votes

Allow setting flow variables in scheduler

Suggested by: Bryan (27 Mar, '20) • Upvoted: 05 Oct, '20 • Comments: 0
Done roadmap
3 votes

Allow deleting multiple transforms from a single flow

Allow deleting multiple transforms from a single flow to make it easier to duplicate a large flow and only keep a small subset of tables from it.
Suggested by: Bryan (09 Mar, '20) • Upvoted: 19 Mar, '20 • Comments: 0
Planned roadmap
3 votes

Search functionality / better documentation of flows

We use a good deal of javascript and in complex flows it can be really difficult to debug, esp. global and flow variables because there's no way to search (that I ...
Suggested by: Shanon (31 Oct, '19) • Upvoted: 19 Mar, '20 • Comments: 0
Planned roadmap
3 votes

Drag&drop SQL query builder

Suggested by: Olena Sherbinin (13 Mar, '19) • Upvoted: 03 Oct, '19 • Comments: 0
Planned roadmap
2 votes

Statistics Page Should include Agent Metrics too

Currently Statistics page does not include metrics for the flows which are executed by Agent... it would make dashboard much better if we can see the whole picture ...
Suggested by: Atul Kapoor (09 May, '22) • Upvoted: 09 May, '22 • Comments: 3
Done roadmap
2 votes

Allow for table list alphabetization in explorer

Expanded salesforce tables in explorer, table names are not alphabetized, makes it hard to find anything
Suggested by: Bryan Meyerovich (10 Jun, '20) • Upvoted: 10 Jun, '20 • Comments: 2
Done roadmap
2 votes

Add ability to generate new Google Sheets

I'm looking for a way to generate new Google Sheets from ETLworks. Or at minimum, use the Google Drive connector to create a new file, then the Google Sheets ...
Suggested by: Frank (30 Nov, '21) • Upvoted: 30 Nov, '21 • Comments: 1
Done roadmap
2 votes

Feature request: smb storage connector through ssh bastion

Suggested by: Bryan (27 Apr, '21) • Upvoted: 27 Apr, '21 • Comments: 1
Done roadmap
2 votes

Add message and script to scheduler email

I want to add a message to the scheduler that contains all the handled flow exceptions. Add a message field where we can append additional text to the default email ...
Suggested by: Bryan (24 Mar, '20) • Upvoted: 05 Oct, '20 • Comments: 1
Planned roadmap
2 votes

Searching flows should also search table names

I want to find a flow that syncs a specific table, the search only includes the flow names. If it's easy include the table names or other sub-fields or scripts
Suggested by: Bryan (12 Feb, '20) • Upvoted: 18 Mar, '20 • Comments: 1
Planned roadmap
2 votes

Implement Kafka Headers

Please consider implementing Kafka Headers. Thanks!
Suggested by: Ryan Cronin (20 Jul, '22) • Upvoted: 20 Jul, '22 • Comments: 0
Done roadmap
2 votes

Add ability to filter flow executions by status

Is it possible to filter events in the flow dashboard by status: success, error, etc?
Suggested by: Olena Sherbinin (15 Aug, '21) • Upvoted: 15 Aug, '21 • Comments: 0
Done roadmap
2 votes

Execute as script in nested SQL loop

I need to execute a proc from a nested loop script but I get “executeQuery only accepts the SELECT statement. Use executeUpdate instead.” Maybe add an ‘execute as ...
Suggested by: Olena Sherbinin (15 Aug, '21) • Upvoted: 15 Aug, '21 • Comments: 0
Done roadmap
2 votes

Connector for Google User Activity API

I want to get some data from the google api: https://analyticsreporting.googleapis.com/v4/userActivity:search via oath2.0 I manage to get this done via postman.
Suggested by: Olena Sherbinin (15 Aug, '21) • Upvoted: 15 Aug, '21 • Comments: 0
Done roadmap
2 votes

Create subfolders when saving CDC events in cloud storage

Can you add the ability to automatically create sub-folders in S3 when saving CDC events?
Suggested by: Olena Sherbinin (15 Aug, '21) • Upvoted: 15 Aug, '21 • Comments: 0
Done roadmap
2 votes

Insert all CDC events into a single topic in the message queue

We are streaming CDC events into the Azure Messages Hubs. Currently, Etlworks creates one topic per database.table. Is it possible to insert all events into a single ...
Suggested by: Olena Sherbinin (15 Aug, '21) • Upvoted: 15 Aug, '21 • Comments: 0
Done roadmap
2 votes

Retain Debezium 'before' element

We are steaming CDC events into the Azure Events Hubs. The events are getting 'flatten' by Etlworks. Is it possible to retain the original structure of the CDC event ...
Suggested by: Olena Sherbinin (15 Aug, '21) • Upvoted: 15 Aug, '21 • Comments: 0
Done roadmap
2 votes

Call Greenplum gpload from Etlworks

We are currently using Greenplum gpload utility to load data into on-premise Greenplum database. Is it possible to integrate it with Etlworks?
Suggested by: Olena Sherbinin (15 Aug, '21) • Upvoted: 15 Aug, '21 • Comments: 0
Done roadmap
2 votes

Improve loading data into Google BigQuery

We are currently using database-to-database flow to load data into Google Query. It works but it is slow. It would be great if you could add BogQuery optimized flows, ...
Suggested by: Olena Sherbinin (15 Aug, '21) • Upvoted: 15 Aug, '21 • Comments: 0
Done roadmap
2 votes

Run as script from explorer

Allow run as script from explorer, for multi-line scripts
Suggested by: Bryan (27 Mar, '21) • Upvoted: 28 Mar, '21 • Comments: 0
Done roadmap
2 votes

Running tasks copy request

How about a copy button for each task to copy the script to clipboard? I’ve had to do that a few times now
Suggested by: Bryan (18 Mar, '21) • Upvoted: 18 Mar, '21 • Comments: 0
Done roadmap
2 votes

View of schedules across all tenants + Last execution duraton

A view were all schedules across all tenants are visible with additional column with last execution duration will help balance load across instance. It will help pick ...
Suggested by: Jaroslaw (20 Nov, '20) • Upvoted: 26 Nov, '20 • Comments: 0
Planned roadmap
2 votes

Add content encoding to gzip files copied to s3

postgres aws_s3.table_import_from_s3 requires content-encoding to be set on gzip files stored in s3. I had to manually add content-encoding header to the s3 ...
Suggested by: Bryan Meyerovich (10 Oct, '20) • Upvoted: 07 Nov, '20 • Comments: 0
Done roadmap
2 votes

Use datetime2 for sql server instead of datetime

Somehow someone in my postgres database was born in the year 1050. Old mofo
Suggested by: Bryan Meyerovich (27 Aug, '20) • Upvoted: 27 Aug, '20 • Comments: 0
Done roadmap
2 votes

Max runtime value for scheduled job

Add a field if scheduled job runs over X minutes automatically kill it. Had a job hang 17 hours, some sort of lock on internal logging I think.
Suggested by: Bryan Meyerovich (03 Jun, '20) • Upvoted: 05 Jun, '20 • Comments: 0
Done roadmap
2 votes

Add a next/previous transform button when editing transform details

Easily transition between transforms to more quickly apply a change to all transforms without losing your place in the list going back and fourth
Suggested by: Bryan (19 Mar, '20) • Upvoted: 19 Mar, '20 • Comments: 0
Planned roadmap
2 votes

UI/UX enhancements

1. By default filter out users from deactivated tenants 2. Add right mouse click in Explorer 3. Do not display "scheduled" icon (or display it as grayed out) for ...
Suggested by: Maksym (03 Oct, '19) • Upvoted: 15 Oct, '19 • Comments: 0
Planned roadmap
2 votes

Code snippets (library)

1. Anywhere within the code editor add a library of reusable code snippets (language dependent) 2. Any saved code snippet can be referenced by name in the actual ...
Suggested by: Olena Sherbinin (06 Jun, '19) • Upvoted: 18 Sep, '19 • Comments: 0
Done roadmap
2 votes

Table view in Test Transformation

1) If the destination connection is a database display output in a table view 2) Use code editor for a preview
Suggested by: Etlworks (10 Jan, '19) • Upvoted: 03 Oct, '19 • Comments: 0
Done roadmap
2 votes

Connectors for consumer cloud storages

Add connectors for consumer cloud storages: - OneDrive and OneDrive for business - Google Drive - Box - DropBox
Suggested by: Etlworks (06 Dec, '18) • Upvoted: 15 Oct, '19 • Comments: 0
Done roadmap
2 votes

Connector for Apache Kafka

Add a read/write connector for Apache Kafka
Suggested by: Etlworks (06 Dec, '18) • Upvoted: 03 Oct, '19 • Comments: 0
Done roadmap
2 votes

Ability to create and share templates

Templates are similar to export/import. Additionally, users should be able to: - Create a template from any Flow/Connection/Format. - Share templates with everyone ...
Suggested by: Etlworks (06 Dec, '18) • Upvoted: 07 Jul, '19 • Comments: 0
Planned roadmap