Our customers made these suggestions for improving Etlworks Integrator. Suggestions for improving Etlworks Integrator.
Packages and versioning
It would be nice to create a "package" of flows and connections and store it as a certain version. It should then be easy to switch between versions just by toggling ...
Done
roadmap
2FA Login
Our security policy will require all tools that have access to company data, to have 2FA required to login.
Currently a user can login using only the username and ...
Done
SSO Authentication
Etlworks handles and processes a lot of sensitive data. It is important that this is adequately shielded and becomes part of our company-wide security umbrella. The ...
Done
roadmap
Email on schedule failure only if last retry
I don't want emails for transient errors if a retry worked
Done
Ability to schedule and execute flows by data integration agents running behind the firewall
Basically, add support for hybrid data integration when flows can be executed on data integration agents running behind the firewall.
Done
roadmap
New fields' mapping editor
Enhance field mapping editor:
- add support for nested fields in the destination
- add an ability to modify attributes, such as JSON array, XML CDATA, XML ...
Planned
roadmap
Better ways to determine how flows, connections, etc. are interconnected
Right now when you have multiple flows that share connections or other flows, it's difficult to understand how making a change to one will possibly impact other ...
Done
roadmap
Implement priority queue and throttling for user-defined APIs (listeners)
Scenario 1: upstream system slammed the system with thousands of API calls in a short span of time, eventually freezing and requiring a reboot to stop all jobs ...
Done
roadmap
Usability enhancement for Flows Builder
1. Ability to edit inner flow from the nested flow
2. Ability to stop API flow, also stop flow from scheduler
3. Add Find Usage for flows (within nested flows)
Planned
roadmap
Drag and drop process builder for nested flows
Replace existing nested flow editor with drag and drop process builder:
- works similarly to the diagram editor
- ability to select existing flows from the side-bar ...
Planned
roadmap
Provisioning from AWS and Azure Marketplaces
Add ability to provision Etlworks from AWS and Azure Marketplaces with Amazon and Microsoft billing
Planned
roadmap
Connector for Google Analytics 4,
Since Google is discontinuing Universal Analytics and moving to only supporting Google Analytics 4,
Done
roadmap
IBM i Debezium Connector
Hello, would it be possible to add IBM i/AS400 Debezium Connector capabilities via this repo please?
https://github.com/jhc-systems/debezium-connector-ibmi
Planned
roadmap
Add an apply to all button when editing a transform
Apply the settings for that transform to all the other transforms in the current flow
Planned
roadmap
Auto-completion in JavaScript code editors
Add auto-completion in JavaScript code editors:
- {field names}
- functions
Planned
roadmap
Scheduler queue
Introduce queuing in scheduler so you can set how many scheduled tasks can execute simultaneously, and if one tries to run while another is running it will wait till ...
Done
roadmap
Allow creation of new connections for each parallel execution in Bulk to Synapse type flows.
Currently, in Nested flows, when executing consecutive SQL Script flows with "Parallel" nested flow option turned on, new connection is created for each SQL Script ...
Done
Bulk insert with db to db flows
Having inserts into postgres and sqlserver work like redshift would be a nice improvement. Right now it runs a bunch of insert statements. Instead, pull source data ...
Done
roadmap
Add ability to resume a sync from where it might have failed, or was stopped/canceled
Sometimes syncs fail for whatever reasons, but being able to easily resume from the flows that were not completed would be fantastic (assuming we separately addressed ...
Done
Import entire schemes without specifying individual table
I really liked how aws dms operated, you pointed it at a schema on your source, told it the target schema in your destination, and had a wildcard supporting ...
Done
roadmap
UI Suggestion: Flow execution/failure duration metrics
Currently, some flows only have a console log of events that happen during flow execution. It would be useful to have a graphical representation, especially nested ...
Done
FLOWs visual based on tree hierarchy
When having multiple flows, it would be nice if you can collapse actions into the nested flows (like a tree).
Result then, only nested flows will be shown, until ...
Under consideration
Add interactive Azure Active directory authentication for SQL Server connector
SQL Server JDBC driver supports accessToken authentication which requires a web flow for signing in with Microsoft. It would be great if Etlworks connector could ...
Done
Support IAM For S3 Authentication
We run ETL Works on a self-hosted EC2 server. To be able to connect to AWS S3 from ETL Works, it currently requires an access key and secret key. Our organisation ...
Done
Allow setting flow parameter in explorer
Made a api connector with a {param1} in the url, want to do this in explorer: select * from content where param1=123
Planned
roadmap
Allow deleting multiple transforms from a single flow
Allow deleting multiple transforms from a single flow to make it easier to duplicate a large flow and only keep a small subset of tables from it.
Planned
roadmap
Search functionality / better documentation of flows
We use a good deal of javascript and in complex flows it can be really difficult to debug, esp. global and flow variables because there's no way to search (that I ...
Planned
roadmap
Drag&drop SQL query builder
Planned
roadmap
Statistics Page Should include Agent Metrics too
Currently Statistics page does not include metrics for the flows which are executed by Agent... it would make dashboard much better if we can see the whole picture ...
Done
roadmap
Add run schedule api endpoint
I'd like to run schedule job via api instead of just flow. This way I can take advantage of the scheduler's extra options like expected flow execution time. Pass ...
Done
Recreating table for HWM flow after data is populated
On a non-HWM transform with new source columns, the destination table isn’t dropped until after the data is populated. On a HWM transform it drops it before data ...
Under consideration
Allow for table list alphabetization in explorer
Expanded salesforce tables in explorer, table names are not alphabetized, makes it hard to find anything
Done
roadmap
Add button to restart service
Sometimes there is a need to restart ETL works service, it would be nice to have a button or something like that in the interface
Not planned
Add ability to generate new Google Sheets
I'm looking for a way to generate new Google Sheets from ETLworks.
Or at minimum, use the Google Drive connector to create a new file, then the Google Sheets ...
Done
roadmap
Parallel CDC snapshot
Currently, CDC flows run snapshots sequentially, which for large databases can be a bottleneck. It would be great to have an option to run snapshots in parallel, each ...
Not planned
support parameterized macros
allow passing params directly to macros when referencing them
{{4:MacroName}('MySpecialParam')}
or something like that
Not planned
Allow setting connections per transform instead of per flow for bulk flows
In my use case I can put my transforms all in one flow, instead of one transform per flow and putting all of them into another nested flow
Done
Change View Schedules Icon if all schedules are disabled
I disabled some schedules and kept some active and realized when i sorted by "Schedules" in the flow list screen they all look the same
Done
Cdata smartsheet data type conversion
I’m doing the text replace in create table sql and it’s working fine, I’ve had to replace datetime and bigint for now. I don’t expect many random new tables to port ...
Done
Add {SCHEMA} to copy sql
I need a parameter I can use in the COPY SQL to inject the target schema. {TABLE} has targetschema.tablename, I just need targetschema
Done
Add an option for redshift destination connections to create all strings as max
So in redshift there is no nvarchar, and varchar(1) is 1 byte, not 1 character. So Unicode strings are getting truncated. I’d prefer all my redshift strings just be ...
Done