After updating the shared libraries, the associated ETL spark job exeuctions are getting failed
Affected Versions: 2.2 2.1 2.0 1.14 1.13
Fix Version: 2.2* 2.1* 2.0* 1.14* 1.13*
Root cause(s)
When updating the shared libraries, the associated ETL spark job attributes are resetting to defaults with version 0.9 in AWS console, which leads to failure of ETL job executions.
List of ETL job attributes resetting are:
- Glue Version
- Max Retries
- Max Concurrent Runs
- Timeout
- Notify Delay After
- Number Of Workers
- Worker Type
Impact
If user updates any shared library which is attached to ETL spark job, the job execution will fail.
Mitigation
Fix available
A fix is available in Amorphic version 2.2. Please upgrade to the latest version to resolve this issue.
Timeline
- 2023-06-16: Bug reported/identified (CLOUD-3539)
- 2023-06-16: Bug triaged
- 2023-06-16: Bug fixed
- 2023-06-17: Testing completed
- 2023-06-19: Fix is available