Our customers made these suggestions for improving Etlworks Integrator. Suggestions for improving Etlworks Integrator.
Packages and versioning
It would be nice to create a "package" of flows and connections and store it as a certain version. It should then be easy to switch between versions just by toggling ...
Done
roadmap
SSO Authentication
Etlworks handles and processes a lot of sensitive data. It is important that this is adequately shielded and becomes part of our company-wide security umbrella. The ...
Done
roadmap
Ability to schedule and execute flows by data integration agents running behind the firewall
Basically, add support for hybrid data integration when flows can be executed on data integration agents running behind the firewall.
Done
roadmap
Usability enhancement for Flows Builder
1. Ability to edit inner flow from the nested flow
2. Ability to stop API flow, also stop flow from scheduler
3. Add Find Usage for flows (within nested flows)
Planned
roadmap
New fields' mapping editor
Enhance field mapping editor:
- add support for nested fields in the destination
- add an ability to modify attributes, such as JSON array, XML CDATA, XML ...
Planned
roadmap
Drag and drop process builder for nested flows
Replace existing nested flow editor with drag and drop process builder:
- works similarly to the diagram editor
- ability to select existing flows from the side-bar ...
Planned
roadmap
Better ways to determine how flows, connections, etc. are interconnected
Right now when you have multiple flows that share connections or other flows, it's difficult to understand how making a change to one will possibly impact other ...
Done
roadmap
Implement priority queue and throttling for user-defined APIs (listeners)
Scenario 1: upstream system slammed the system with thousands of API calls in a short span of time, eventually freezing and requiring a reboot to stop all jobs ...
Done
roadmap
Provisioning from AWS and Azure Marketplaces
Add ability to provision Etlworks from AWS and Azure Marketplaces with Amazon and Microsoft billing
Planned
roadmap
Connector for Google Analytics 4,
Since Google is discontinuing Universal Analytics and moving to only supporting Google Analytics 4,
Done
roadmap
Search functionality / better documentation of flows
We use a good deal of javascript and in complex flows it can be really difficult to debug, esp. global and flow variables because there's no way to search (that I ...
Done
roadmap
IBM i Debezium Connector
Hello, would it be possible to add IBM i/AS400 Debezium Connector capabilities via this repo please?
https://github.com/jhc-systems/debezium-connector-ibmi
Planned
roadmap
Add an apply to all button when editing a transform
Apply the settings for that transform to all the other transforms in the current flow
Planned
roadmap
Drag&drop SQL query builder
Planned
roadmap
Auto-completion in JavaScript code editors
Add auto-completion in JavaScript code editors:
- {field names}
- functions
Planned
roadmap
Scheduler queue
Introduce queuing in scheduler so you can set how many scheduled tasks can execute simultaneously, and if one tries to run while another is running it will wait till ...
Done
roadmap
Bulk insert with db to db flows
Having inserts into postgres and sqlserver work like redshift would be a nice improvement. Right now it runs a bunch of insert statements. Instead, pull source data ...
Done
roadmap
Import entire schemes without specifying individual table
I really liked how aws dms operated, you pointed it at a schema on your source, told it the target schema in your destination, and had a wildcard supporting ...
Done
roadmap
Allow setting flow parameter in explorer
Made a api connector with a {param1} in the url, want to do this in explorer: select * from content where param1=123
Planned
roadmap
Allow deleting multiple transforms from a single flow
Allow deleting multiple transforms from a single flow to make it easier to duplicate a large flow and only keep a small subset of tables from it.
Planned
roadmap
Statistics Page Should include Agent Metrics too
Currently Statistics page does not include metrics for the flows which are executed by Agent... it would make dashboard much better if we can see the whole picture ...
Done
roadmap
Allow for table list alphabetization in explorer
Expanded salesforce tables in explorer, table names are not alphabetized, makes it hard to find anything
Done
roadmap
Add ability to generate new Google Sheets
I'm looking for a way to generate new Google Sheets from ETLworks.
Or at minimum, use the Google Drive connector to create a new file, then the Google Sheets ...
Done
roadmap
Add message and script to scheduler email
I want to add a message to the scheduler that contains all the handled flow exceptions.
Add a message field where we can append additional text to the default email ...
Planned
roadmap
Searching flows should also search table names
I want to find a flow that syncs a specific table, the search only includes the flow names. If it's easy include the table names or other sub-fields or scripts
Planned
roadmap
Add ability to filter flow executions by status
Is it possible to filter events in the flow dashboard by status: success, error, etc?
Done
roadmap
Execute as script in nested SQL loop
I need to execute a proc from a nested loop script but I get “executeQuery only accepts the SELECT statement. Use executeUpdate instead.”
Maybe add an ‘execute as ...
Done
roadmap
Connector for Google User Activity API
I want to get some data from the google api: https://analyticsreporting.googleapis.com/v4/userActivity:search via oath2.0
I manage to get this done via postman.
Done
roadmap
Create subfolders when saving CDC events in cloud storage
Can you add the ability to automatically create sub-folders in S3 when saving CDC events?
Done
roadmap
Insert all CDC events into a single topic in the message queue
We are streaming CDC events into the Azure Messages Hubs. Currently, Etlworks creates one topic per database.table. Is it possible to insert all events into a single ...
Done
roadmap
Retain Debezium 'before' element
We are steaming CDC events into the Azure Events Hubs. The events are getting 'flatten' by Etlworks. Is it possible to retain the original structure of the CDC event ...
Done
roadmap
Call Greenplum gpload from Etlworks
We are currently using Greenplum gpload utility to load data into on-premise Greenplum database. Is it possible to integrate it with Etlworks?
Done
roadmap
Improve loading data into Google BigQuery
We are currently using database-to-database flow to load data into Google Query. It works but it is slow. It would be great if you could add BogQuery optimized flows, ...
Done
roadmap
Running tasks copy request
How about a copy button for each task to copy the script to clipboard? I’ve had to do that a few times now
Done
roadmap
View of schedules across all tenants + Last execution duraton
A view were all schedules across all tenants are visible with additional column with last execution duration will help balance load across instance. It will help pick ...
Planned
roadmap
Add content encoding to gzip files copied to s3
postgres aws_s3.table_import_from_s3 requires content-encoding to be set on gzip files stored in s3. I had to manually add content-encoding header to the s3 ...
Done
roadmap
Use datetime2 for sql server instead of datetime
Somehow someone in my postgres database was born in the year 1050. Old mofo
Done
roadmap
Max runtime value for scheduled job
Add a field if scheduled job runs over X minutes automatically kill it. Had a job hang 17 hours, some sort of lock on internal logging I think.
Done
roadmap
Add a next/previous transform button when editing transform details
Easily transition between transforms to more quickly apply a change to all transforms without losing your place in the list going back and fourth
Planned
roadmap
UI/UX enhancements
1. By default filter out users from deactivated tenants
2. Add right mouse click in Explorer
3. Do not display "scheduled" icon (or display it as grayed out) for ...
Planned
roadmap
Code snippets (library)
1. Anywhere within the code editor add a library of reusable code snippets (language dependent)
2. Any saved code snippet can be referenced by name in the actual ...
Done
roadmap
Table view in Test Transformation
1) If the destination connection is a database display output in a table view
2) Use code editor for a preview
Done
roadmap
Connectors for consumer cloud storages
Add connectors for consumer cloud storages:
- OneDrive and OneDrive for business
- Google Drive
- Box
- DropBox
Done
roadmap