getting error while executing airflow dag in transform seed in submit errors.csv
757061562299,[2025-09-05T08:39:22.299+0000] {elephant_workflow.py:91} ERROR - Validation errors detected:
1757061562299,"[2025-09-05T08:39:22.299+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""missing required property 'json'"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""missing required property 'body'"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to/source_http_request"", ""error_message"": ""missing required property 'json'"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to/source_http_request"", ""error_message"": ""missing required property 'body'"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""missing required property 'json'"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""missing required property 'body'"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""missing required property 'json'"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""missing required property 'body'"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562304,[2025-09-05T08:39:22.304+0000] {elephant_workflow.py:799} ERROR - Error in transform_seed: Validation failed: submit_errors.csv contains errors
1757061562304,[2025-09-05T08:39:22.304+0000] {elephant_workflow.py:125} INFO - Uploading to S3: /tmp/tmp55y4yim1/error.json to s3://elephant-oracle-node-environmentbucket-e7zbxflbv5ek/outputs/2025-09-05T08:36:02.208801+00:00/1cdd0ec1/474128015420.csv/errors/transform_seed_error.json
1757061562393,"[2025-09-05T08:39:22.393+0000] {taskinstance.py:3311} ERROR - Task failed with exception
Traceback (most recent call last):
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 762, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 733, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py"", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py"", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/decorators/base.py"", line 266, in execute
return_value = super().execute(context)
^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py"", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py"", line 238, in execute
return_value = self.execute_callable()
^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py"", line 256, in execute_callable
return runner.run(*self.op_args, **self.op_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py"", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/dags/elephant_workflow.py"", line 775, in transform_seed
run_command(cmd, timeout=300, cwd=str(tmp_path))
File ""/usr/local/airflow/dags/elephant_workflow.py"", line 94, in run_command
raise Exception(""Validation failed: submit_errors.csv contains errors"")
Exception: Validation failed: submit_errors.csv contains errors"
1757061562412,"[2025-09-05T08:39:22.412+0000] {taskinstance.py:1225} INFO - Marking task as FAILED. dag_id=elephant_workflow, task_id=process_single_file.transform_seed, run_id=scheduled__2025-09-05T08:35:00+00:00, map_index=6, execution_date=20250905T083500, start_date=20250905T083841, end_date=20250905T083922"
1757061562439,[2025-09-05T08:39:22.439+0000] {taskinstance.py:340} INFO - ::group::Post task execution logs
1757061562440,"[2025-09-05T08:39:22.440+0000] {standard_task_runner.py:124} ERROR - Failed to execute job 2538 for task process_single_file.transform_seed (Validation failed: submit_errors.csv contains errors; 9902)
Traceback (most recent call last):
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/task/task_runner/standard_task_runner.py"", line 117, in _start_by_fork
ret = args.func(args, dag=self.dag)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/cli_config.py"", line 49, in command
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/cli.py"", line 116, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/task_command.py"", line 483, in task_run
task_return_code = _run_task_by_selected_method(args, _dag, ti)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/task_command.py"", line 256, in _run_task_by_selected_method
return _run_raw_task(args, ti)
^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/task_command.py"", line 341, in _run_raw_task
return ti._run_raw_task(
^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/session.py"", line 97, in wrapper
return func(*args, session=session, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 3005, in _run_raw_task
return _run_raw_task(
^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 273, in _run_raw_task
TaskInstance._execute_task_with_callbacks(
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 3159, in _execute_task_with_callbacks
result = self._execute_task(context, task_orig)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 3183, in _execute_task
return _execute_task(self, context, task_orig)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 762, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 733, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py"", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py"", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/decorators/base.py"", line 266, in execute
return_value = super().execute(context)
^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py"", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py"", line 238, in execute
return_value = self.execute_callable()
^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py"", line 256, in execute_callable
return runner.run(*self.op_args, **self.op_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py"", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/dags/elephant_workflow.py"", line 775, in transform_seed
run_command(cmd, timeout=300, cwd=str(tmp_path))
File ""/usr/local/airflow/dags/elephant_workflow.py"", line 94, in run_command
raise Exception(""Validation failed: submit_errors.csv contains errors"")
Exception: Validation failed: submit_errors.csv contains errors"
1757061562659,[2025-09-05T08:39:22.659+0000] {local_task_job_runner.py:266} INFO - Task exited with return code 1
1757061562659,[2025-09-05T08:39:22.659+0000] {local_task_job_runner.py:245} INFO - ::endgroup::
i guess for me it is the POST request and json payload and in property seed it is not matching correctly whatever is there in source_http_request
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
QUESTIONS:-
should i change the script in dag or any other way how it should be done
how can we access everything from /tmp location for debugging purpose
getting error while executing airflow dag in transform seed in submit errors.csv
757061562299,[2025-09-05T08:39:22.299+0000] {elephant_workflow.py:91} ERROR - Validation errors detected:
1757061562299,"[2025-09-05T08:39:22.299+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""missing required property 'json'"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""missing required property 'body'"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to/source_http_request"", ""error_message"": ""missing required property 'json'"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to/source_http_request"", ""error_message"": ""missing required property 'body'"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562301,"[2025-09-05T08:39:22.301+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/to"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""missing required property 'json'"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""missing required property 'body'"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562302,"[2025-09-05T08:39:22.302+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed_has_fact_sheet/0/from"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""missing required property 'json'"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""missing required property 'body'"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562303,"[2025-09-05T08:39:22.303+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/unnormalized_address_has_fact_sheet/0/from"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.265Z""}"
1757061562304,[2025-09-05T08:39:22.304+0000] {elephant_workflow.py:799} ERROR - Error in transform_seed: Validation failed: submit_errors.csv contains errors
1757061562304,[2025-09-05T08:39:22.304+0000] {elephant_workflow.py:125} INFO - Uploading to S3: /tmp/tmp55y4yim1/error.json to s3://elephant-oracle-node-environmentbucket-e7zbxflbv5ek/outputs/2025-09-05T08:36:02.208801+00:00/1cdd0ec1/474128015420.csv/errors/transform_seed_error.json
1757061562393,"[2025-09-05T08:39:22.393+0000] {taskinstance.py:3311} ERROR - Task failed with exception
Traceback (most recent call last):
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 762, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 733, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py"", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py"", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/decorators/base.py"", line 266, in execute
return_value = super().execute(context)
^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py"", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py"", line 238, in execute
return_value = self.execute_callable()
^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py"", line 256, in execute_callable
return runner.run(*self.op_args, **self.op_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py"", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/dags/elephant_workflow.py"", line 775, in transform_seed
run_command(cmd, timeout=300, cwd=str(tmp_path))
File ""/usr/local/airflow/dags/elephant_workflow.py"", line 94, in run_command
raise Exception(""Validation failed: submit_errors.csv contains errors"")
Exception: Validation failed: submit_errors.csv contains errors"
1757061562412,"[2025-09-05T08:39:22.412+0000] {taskinstance.py:1225} INFO - Marking task as FAILED. dag_id=elephant_workflow, task_id=process_single_file.transform_seed, run_id=scheduled__2025-09-05T08:35:00+00:00, map_index=6, execution_date=20250905T083500, start_date=20250905T083841, end_date=20250905T083922"
1757061562439,[2025-09-05T08:39:22.439+0000] {taskinstance.py:340} INFO - ::group::Post task execution logs
1757061562440,"[2025-09-05T08:39:22.440+0000] {standard_task_runner.py:124} ERROR - Failed to execute job 2538 for task process_single_file.transform_seed (Validation failed: submit_errors.csv contains errors; 9902)
Traceback (most recent call last):
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/task/task_runner/standard_task_runner.py"", line 117, in _start_by_fork
ret = args.func(args, dag=self.dag)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/cli_config.py"", line 49, in command
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/cli.py"", line 116, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/task_command.py"", line 483, in task_run
task_return_code = _run_task_by_selected_method(args, _dag, ti)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/task_command.py"", line 256, in _run_task_by_selected_method
return _run_raw_task(args, ti)
^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/task_command.py"", line 341, in _run_raw_task
return ti._run_raw_task(
^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/session.py"", line 97, in wrapper
return func(*args, session=session, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 3005, in _run_raw_task
return _run_raw_task(
^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 273, in _run_raw_task
TaskInstance._execute_task_with_callbacks(
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 3159, in _execute_task_with_callbacks
result = self._execute_task(context, task_orig)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 3183, in _execute_task
return _execute_task(self, context, task_orig)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 762, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py"", line 733, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py"", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py"", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/decorators/base.py"", line 266, in execute
return_value = super().execute(context)
^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py"", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py"", line 238, in execute
return_value = self.execute_callable()
^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py"", line 256, in execute_callable
return runner.run(*self.op_args, **self.op_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py"", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File ""/usr/local/airflow/dags/elephant_workflow.py"", line 775, in transform_seed
run_command(cmd, timeout=300, cwd=str(tmp_path))
File ""/usr/local/airflow/dags/elephant_workflow.py"", line 94, in run_command
raise Exception(""Validation failed: submit_errors.csv contains errors"")
Exception: Validation failed: submit_errors.csv contains errors"
1757061562659,[2025-09-05T08:39:22.659+0000] {local_task_job_runner.py:266} INFO - Task exited with return code 1
1757061562659,[2025-09-05T08:39:22.659+0000] {local_task_job_runner.py:245} INFO - ::endgroup::
i guess for me it is the POST request and json payload and in property seed it is not matching correctly whatever is there in source_http_request
1757061562300,"[2025-09-05T08:39:22.300+0000] {elephant_workflow.py:93} ERROR - {""property_cid"": ""data"", ""data_group_cid"": ""bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe"", ""file_path"": ""/tmp/elephant-cli-zip-SrKVLL/data/bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe.json"", ""error_path"": ""/relationships/property_seed/from/source_http_request"", ""error_message"": ""must match ""then"" schema"", ""timestamp"": ""2025-09-05T08:39:22.264Z""}"
QUESTIONS:-
should i change the script in dag or any other way how it should be done
how can we access everything from /tmp location for debugging purpose