ankitm/airflow_project
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

1st commit

Browse Source
This commit is contained in:
Ankit Malik
2026-06-25 11:20:22 +05:30
parent 6b2d754981
commit 1d5ad2d793
14 changed files with 2322 additions and 620 deletions
Download Patch File Download Diff File Expand all files Collapse all files
logs/etl_20260624.log
+240
View File
@@ -0,0 +1,240 @@
2026-06-24 15:55:46 | INFO | Pipeline Start Date: 2026-06-23
2026-06-24 15:55:46 | INFO | ================================================================================
2026-06-24 15:55:46 | INFO | Hello from data-move Python data pipeline!
2026-06-24 15:55:46 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 15:55:46 | INFO | Connecting to databases...
2026-06-24 15:55:48 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8151D4730>
2026-06-24 15:55:52 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8162F0B90>
2026-06-24 15:55:52 | INFO | Database connections established
2026-06-24 15:55:52 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 15:55:52 | INFO | Found 863 MIDs
2026-06-24 15:55:52 | WARNING | Pipeline failed. Retry 1/3. Error: 'tables'
2026-06-24 15:55:57 | INFO | ================================================================================
2026-06-24 15:55:57 | INFO | Hello from data-move Python data pipeline!
2026-06-24 15:55:57 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 15:55:57 | INFO | Connecting to databases...
2026-06-24 15:55:58 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C81600B550>
2026-06-24 15:55:58 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8162F5A90>
2026-06-24 15:55:59 | INFO | Database connections established
2026-06-24 15:55:59 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 15:55:59 | INFO | Found 863 MIDs
2026-06-24 15:55:59 | WARNING | Pipeline failed. Retry 2/3. Error: 'tables'
2026-06-24 15:56:04 | INFO | ================================================================================
2026-06-24 15:56:04 | INFO | Hello from data-move Python data pipeline!
2026-06-24 15:56:04 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 15:56:04 | INFO | Connecting to databases...
2026-06-24 15:56:04 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8162F59A0>
2026-06-24 15:56:05 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8162F68A0>
2026-06-24 15:56:05 | INFO | Database connections established
2026-06-24 15:56:05 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 15:56:05 | INFO | Found 863 MIDs
2026-06-24 16:10:30 | INFO | Pipeline Start Date: 2026-06-23
2026-06-24 16:10:30 | INFO | ================================================================================
2026-06-24 16:10:30 | INFO | Hello from data-move Python data pipeline!
2026-06-24 16:10:30 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:10:30 | INFO | Connecting to databases...
2026-06-24 16:10:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29EE8C470>
2026-06-24 16:10:32 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FEBCA50>
2026-06-24 16:10:32 | INFO | Database connections established
2026-06-24 16:10:32 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 16:10:33 | INFO | Found 863 MIDs
2026-06-24 16:10:33 | WARNING | Pipeline failed. Retry 1/3. Error: 'tables'
2026-06-24 16:10:38 | INFO | ================================================================================
2026-06-24 16:10:38 | INFO | Hello from data-move Python data pipeline!
2026-06-24 16:10:38 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:10:38 | INFO | Connecting to databases...
2026-06-24 16:10:38 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FBB7550>
2026-06-24 16:10:39 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FECD9A0>
2026-06-24 16:10:40 | INFO | Database connections established
2026-06-24 16:10:40 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 16:10:40 | INFO | Found 863 MIDs
2026-06-24 16:10:40 | WARNING | Pipeline failed. Retry 2/3. Error: 'tables'
2026-06-24 16:10:45 | INFO | ================================================================================
2026-06-24 16:10:45 | INFO | Hello from data-move Python data pipeline!
2026-06-24 16:10:45 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:10:45 | INFO | Connecting to databases...
2026-06-24 16:10:45 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FECD8B0>
2026-06-24 16:10:46 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FECE7B0>
2026-06-24 16:10:46 | INFO | Database connections established
2026-06-24 16:10:46 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 16:10:47 | INFO | Found 863 MIDs
2026-06-24 16:12:02 | INFO | Pipeline Start Date: 2026-06-23
2026-06-24 16:12:02 | INFO | ================================================================================
2026-06-24 16:12:02 | INFO | Hello from data-move Python data pipeline!
2026-06-24 16:12:02 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:12:02 | INFO | Connecting to databases...
2026-06-24 16:12:02 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DBAE05D0>
2026-06-24 16:12:03 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB8CA50>
2026-06-24 16:12:04 | INFO | Database connections established
2026-06-24 16:12:04 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 16:12:04 | INFO | Found 863 MIDs
2026-06-24 16:12:05 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:12:05 | INFO | ================================================================================
2026-06-24 16:12:05 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-24 16:12:05 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-24 16:12:05 | INFO | Exists: True
2026-06-24 16:12:05 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
2026-06-24 16:12:05 | INFO | Fetching data for 863 EMPIDs
2026-06-24 16:12:05 | INFO | Fetching OQaD data for run_date=2026-06-23
2026-06-24 16:12:11 | INFO | fn name is fetch_OQad ------Fetched 458 rows
2026-06-24 16:12:11 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
2026-06-24 16:12:11 | INFO | Fetched 458 rows
2026-06-24 16:12:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
2026-06-24 16:12:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-24 16:12:11 | ERROR | Failed processing table OQaD
Traceback (most recent call last):
File "D:\z\airflow-project\elt.py", line 243, in elt
load_to_clickhouse(
~~~~~~~~~~~~~~~~~~^
client=client,
^^^^^^^^^^^^^^
table_name=table_name,
^^^^^^^^^^^^^^^^^^^^^^
df=df,
^^^^^^
)
^
File "D:\z\airflow-project\clickhouse_task\load_table.py", line 28, in load_to_clickhouse
chunk_df = df.iloc[start:end]
^^^^^^^
AttributeError: 'DataFrame' object has no attribute 'iloc'
2026-06-24 16:12:11 | WARNING | Pipeline failed. Retry 1/3. Error: 'DataFrame' object has no attribute 'iloc'
2026-06-24 16:12:16 | INFO | ================================================================================
2026-06-24 16:12:16 | INFO | Hello from data-move Python data pipeline!
2026-06-24 16:12:16 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:12:16 | INFO | Connecting to databases...
2026-06-24 16:12:17 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB6C650>
2026-06-24 16:12:17 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB99A90>
2026-06-24 16:12:18 | INFO | Database connections established
2026-06-24 16:12:18 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 16:12:18 | INFO | Found 863 MIDs
2026-06-24 16:12:18 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:12:18 | INFO | ================================================================================
2026-06-24 16:12:18 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-24 16:12:18 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-24 16:12:18 | INFO | Exists: True
2026-06-24 16:12:18 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
2026-06-24 16:12:18 | INFO | Fetching data for 863 EMPIDs
2026-06-24 16:12:18 | INFO | Fetching OQaD data for run_date=2026-06-23
2026-06-24 16:12:24 | INFO | fn name is fetch_OQad ------Fetched 458 rows
2026-06-24 16:12:24 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
2026-06-24 16:12:24 | INFO | Fetched 458 rows
2026-06-24 16:12:24 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
2026-06-24 16:12:24 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-24 16:12:24 | ERROR | Failed processing table OQaD
Traceback (most recent call last):
File "D:\z\airflow-project\elt.py", line 243, in elt
load_to_clickhouse(
~~~~~~~~~~~~~~~~~~^
client=client,
^^^^^^^^^^^^^^
table_name=table_name,
^^^^^^^^^^^^^^^^^^^^^^
df=df,
^^^^^^
)
^
File "D:\z\airflow-project\clickhouse_task\load_table.py", line 28, in load_to_clickhouse
chunk_df = df.iloc[start:end]
^^^^^^^
AttributeError: 'DataFrame' object has no attribute 'iloc'
2026-06-24 16:12:24 | WARNING | Pipeline failed. Retry 2/3. Error: 'DataFrame' object has no attribute 'iloc'
2026-06-24 16:12:29 | INFO | ================================================================================
2026-06-24 16:12:29 | INFO | Hello from data-move Python data pipeline!
2026-06-24 16:12:29 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:12:29 | INFO | Connecting to databases...
2026-06-24 16:12:30 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB9A120>
2026-06-24 16:12:30 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB9A8A0>
2026-06-24 16:12:31 | INFO | Database connections established
2026-06-24 16:12:31 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 16:12:31 | INFO | Found 863 MIDs
2026-06-24 16:12:31 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:12:31 | INFO | ================================================================================
2026-06-24 16:12:31 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-24 16:12:31 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-24 16:12:31 | INFO | Exists: True
2026-06-24 16:12:31 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
2026-06-24 16:12:31 | INFO | Fetching data for 863 EMPIDs
2026-06-24 16:12:31 | INFO | Fetching OQaD data for run_date=2026-06-23
2026-06-24 16:12:43 | INFO | fn name is fetch_OQad ------Fetched 458 rows
2026-06-24 16:12:43 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
2026-06-24 16:12:43 | INFO | Fetched 458 rows
2026-06-24 16:12:43 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
2026-06-24 16:12:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-24 16:12:43 | ERROR | Failed processing table OQaD
Traceback (most recent call last):
File "D:\z\airflow-project\elt.py", line 243, in elt
load_to_clickhouse(
~~~~~~~~~~~~~~~~~~^
client=client,
^^^^^^^^^^^^^^
table_name=table_name,
^^^^^^^^^^^^^^^^^^^^^^
df=df,
^^^^^^
)
^
File "D:\z\airflow-project\clickhouse_task\load_table.py", line 28, in load_to_clickhouse
chunk_df = df.iloc[start:end]
^^^^^^^
AttributeError: 'DataFrame' object has no attribute 'iloc'
2026-06-24 16:17:01 | INFO | Pipeline Start Date: 2026-06-23
2026-06-24 16:17:01 | INFO | ================================================================================
2026-06-24 16:17:01 | INFO | Hello from data-move Python data pipeline!
2026-06-24 16:17:01 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:17:01 | INFO | Connecting to databases...
2026-06-24 16:17:02 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000018D3B1F05D0>
2026-06-24 16:17:03 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000018D3C29CA50>
2026-06-24 16:17:04 | INFO | Database connections established
2026-06-24 16:17:04 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 16:17:04 | INFO | Found 863 MIDs
2026-06-24 16:17:04 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:17:04 | INFO | ================================================================================
2026-06-24 16:17:04 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-24 16:17:04 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-24 16:17:04 | INFO | Exists: True
2026-06-24 16:17:04 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
2026-06-24 16:17:04 | INFO | Fetching data for 863 EMPIDs
2026-06-24 16:17:04 | INFO | Fetching OQaD data for run_date=2026-06-23
2026-06-24 16:17:10 | INFO | fn name is fetch_OQad ------Fetched 458 rows
2026-06-24 16:17:10 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
2026-06-24 16:17:10 | INFO | Fetched 458 rows
2026-06-24 16:17:10 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
2026-06-24 16:17:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-24 16:17:11 | INFO | Inserted rows 0 to 458
2026-06-24 16:17:11 | INFO | OQaD: inserted 458 rows into ClickHouse
2026-06-24 16:17:11 | INFO | OQaD loaded successfully (458 rows)
2026-06-24 16:17:11 | INFO | ================================================================================
2026-06-24 16:17:11 | INFO | Pipeline Completed Successfully
2026-06-24 16:17:11 | INFO | ================================================================================
2026-06-24 16:17:11 | INFO | Pipeline completed successfully. pipeline_trigeered_on_date=2026-06-24last_successful_run_date=2026-06-23
2026-06-24 16:17:31 | INFO | Pipeline Start Date: 2026-06-23
2026-06-24 16:17:31 | INFO | ================================================================================
2026-06-24 16:17:31 | INFO | Hello from data-move Python data pipeline!
2026-06-24 16:17:31 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:17:31 | INFO | Connecting to databases...
2026-06-24 16:17:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000022F667F05D0>
2026-06-24 16:17:33 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000022F678CCA50>
2026-06-24 16:17:33 | INFO | Database connections established
2026-06-24 16:17:33 | INFO | Collecting MIDs for: 2026-06-23
2026-06-24 16:17:34 | INFO | Found 863 MIDs
2026-06-24 16:17:34 | INFO | Pipeline Run Date: 2026-06-23
2026-06-24 16:17:34 | INFO | ================================================================================
2026-06-24 16:17:34 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-24 16:17:34 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-24 16:17:34 | INFO | Exists: True
2026-06-24 16:17:34 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
2026-06-24 16:17:34 | INFO | Fetching data for 863 EMPIDs
2026-06-24 16:17:34 | INFO | Fetching OQaD data for run_date=2026-06-23
2026-06-24 16:17:40 | INFO | fn name is fetch_OQad ------Fetched 458 rows
2026-06-24 16:17:40 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
2026-06-24 16:17:40 | INFO | Fetched 458 rows
2026-06-24 16:17:40 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
2026-06-24 16:17:41 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-24 16:17:41 | INFO | Inserted rows 0 to 458
2026-06-24 16:17:41 | INFO | OQaD: inserted 458 rows into ClickHouse
2026-06-24 16:17:41 | INFO | OQaD loaded successfully (458 rows)
2026-06-24 16:17:41 | INFO | ================================================================================
2026-06-24 16:17:41 | INFO | Pipeline Completed Successfully
2026-06-24 16:17:41 | INFO | ================================================================================
2026-06-24 16:17:41 | INFO | Pipeline completed successfully. pipeline_trigeered_on_date=2026-06-24last_successful_run_date=2026-06-23
logs/etl_20260625.log
+90
View File
@@ -0,0 +1,90 @@
2026-06-25 10:23:52 | INFO | Pipeline Start Date: 2026-06-24
2026-06-25 10:23:52 | INFO | ================================================================================
2026-06-25 10:23:52 | INFO | Hello from data-move Python data pipeline!
2026-06-25 10:23:52 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:23:52 | INFO | Connecting to databases...
2026-06-25 10:23:54 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D3489C90D0>
2026-06-25 10:23:57 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349ACA490>
2026-06-25 10:23:58 | INFO | Database connections established
2026-06-25 10:23:58 | INFO | Collecting MIDs for: 2026-06-24
2026-06-25 10:23:58 | INFO | Found 868 MIDs
2026-06-25 10:23:59 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:23:59 | INFO | ================================================================================
2026-06-25 10:23:59 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-25 10:23:59 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-25 10:23:59 | ERROR | Failed processing table OQaD
Traceback (most recent call last):
File "D:\z\airflow-project\elt2.py", line 172, in elt
df=registry[f"fetch_{table_name}"]()(sql_engine, table_name , table_type, mids, run_date)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
TypeError: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 10:23:59 | WARNING | Pipeline failed. Retry 1/3. Error: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 10:24:04 | INFO | ================================================================================
2026-06-25 10:24:04 | INFO | Hello from data-move Python data pipeline!
2026-06-25 10:24:04 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:24:04 | INFO | Connecting to databases...
2026-06-25 10:24:05 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349AC3950>
2026-06-25 10:24:05 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349B68E60>
2026-06-25 10:24:06 | INFO | Database connections established
2026-06-25 10:24:06 | INFO | Collecting MIDs for: 2026-06-24
2026-06-25 10:24:06 | INFO | Found 868 MIDs
2026-06-25 10:24:06 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:24:06 | INFO | ================================================================================
2026-06-25 10:24:06 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-25 10:24:06 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-25 10:24:06 | ERROR | Failed processing table OQaD
Traceback (most recent call last):
File "D:\z\airflow-project\elt2.py", line 172, in elt
df=registry[f"fetch_{table_name}"]()(sql_engine, table_name , table_type, mids, run_date)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
TypeError: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 10:24:06 | WARNING | Pipeline failed. Retry 2/3. Error: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 10:24:11 | INFO | ================================================================================
2026-06-25 10:24:11 | INFO | Hello from data-move Python data pipeline!
2026-06-25 10:24:11 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:24:11 | INFO | Connecting to databases...
2026-06-25 10:24:12 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349B68D70>
2026-06-25 10:24:12 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349B69C70>
2026-06-25 10:24:12 | INFO | Database connections established
2026-06-25 10:24:12 | INFO | Collecting MIDs for: 2026-06-24
2026-06-25 10:24:13 | INFO | Found 868 MIDs
2026-06-25 10:24:13 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:24:13 | INFO | ================================================================================
2026-06-25 10:24:13 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-25 10:24:13 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-25 10:24:13 | ERROR | Failed processing table OQaD
Traceback (most recent call last):
File "D:\z\airflow-project\elt2.py", line 172, in elt
df=registry[f"fetch_{table_name}"]()(sql_engine, table_name , table_type, mids, run_date)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
TypeError: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 11:19:10 | INFO | Pipeline Start Date: 2026-06-24
2026-06-25 11:19:10 | INFO | ================================================================================
2026-06-25 11:19:10 | INFO | Hello from data-move Python data pipeline!
2026-06-25 11:19:10 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 11:19:10 | INFO | Connecting to databases...
2026-06-25 11:19:11 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002B17CBD0890>
2026-06-25 11:19:13 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002B17DCE0910>
2026-06-25 11:19:14 | INFO | Database connections established
2026-06-25 11:19:14 | INFO | Collecting MIDs for: 2026-06-24
2026-06-25 11:19:14 | INFO | Found 868 MIDs
2026-06-25 11:19:15 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 11:19:15 | INFO | ================================================================================
2026-06-25 11:19:15 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-25 11:19:15 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-25 11:19:15 | INFO | Exists: True
2026-06-25 11:19:15 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
2026-06-25 11:19:15 | INFO | Fetching data for 868 EMPIDs
2026-06-25 11:19:15 | INFO | Fetching OQaD data for run_date=2026-06-24
2026-06-25 11:19:21 | INFO | fn name is fetch_OQad ------Fetched 469 rows
2026-06-25 11:19:21 | INFO | Fetched total row -: 469 from sql server for table-:OQaD ...........!!!
2026-06-25 11:19:21 | INFO | Fetched 469 rows
2026-06-25 11:19:21 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
2026-06-25 11:19:21 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-25 11:19:21 | INFO | Inserted rows 0 to 469
2026-06-25 11:19:21 | INFO | OQaD: inserted 469 rows into ClickHouse
2026-06-25 11:19:21 | INFO | OQaD loaded successfully (469 rows)
2026-06-25 11:19:21 | INFO | ================================================================================
2026-06-25 11:19:21 | INFO | Pipeline Completed Successfully
2026-06-25 11:19:21 | INFO | ================================================================================
2026-06-25 11:19:21 | INFO | Pipeline completed successfully. pipeline_trigeered_on_date=2026-06-25last_successful_run_date=2026-06-24