ankitm/airflow_project
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
test
airflow_project/logs/etl_20260625.log
T
Ankit Malik 1d5ad2d793 1st commit
2026-06-25 11:20:22 +05:30

91 lines
7.3 KiB
Plaintext
Raw Permalink Blame History

2026-06-25 10:23:52 | INFO | Pipeline Start Date: 2026-06-24
2026-06-25 10:23:52 | INFO | ================================================================================
2026-06-25 10:23:52 | INFO | Hello from data-move Python data pipeline!
2026-06-25 10:23:52 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:23:52 | INFO | Connecting to databases...
2026-06-25 10:23:54 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D3489C90D0>
2026-06-25 10:23:57 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349ACA490>
2026-06-25 10:23:58 | INFO | Database connections established
2026-06-25 10:23:58 | INFO | Collecting MIDs for: 2026-06-24
2026-06-25 10:23:58 | INFO | Found 868 MIDs
2026-06-25 10:23:59 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:23:59 | INFO | ================================================================================
2026-06-25 10:23:59 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-25 10:23:59 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-25 10:23:59 | ERROR | Failed processing table OQaD
Traceback (most recent call last):
File "D:\z\airflow-project\elt2.py", line 172, in elt
df=registry[f"fetch_{table_name}"]()(sql_engine, table_name , table_type, mids, run_date)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
TypeError: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 10:23:59 | WARNING | Pipeline failed. Retry 1/3. Error: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 10:24:04 | INFO | ================================================================================
2026-06-25 10:24:04 | INFO | Hello from data-move Python data pipeline!
2026-06-25 10:24:04 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:24:04 | INFO | Connecting to databases...
2026-06-25 10:24:05 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349AC3950>
2026-06-25 10:24:05 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349B68E60>
2026-06-25 10:24:06 | INFO | Database connections established
2026-06-25 10:24:06 | INFO | Collecting MIDs for: 2026-06-24
2026-06-25 10:24:06 | INFO | Found 868 MIDs
2026-06-25 10:24:06 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:24:06 | INFO | ================================================================================
2026-06-25 10:24:06 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-25 10:24:06 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-25 10:24:06 | ERROR | Failed processing table OQaD
Traceback (most recent call last):
File "D:\z\airflow-project\elt2.py", line 172, in elt
df=registry[f"fetch_{table_name}"]()(sql_engine, table_name , table_type, mids, run_date)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
TypeError: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 10:24:06 | WARNING | Pipeline failed. Retry 2/3. Error: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 10:24:11 | INFO | ================================================================================
2026-06-25 10:24:11 | INFO | Hello from data-move Python data pipeline!
2026-06-25 10:24:11 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:24:11 | INFO | Connecting to databases...
2026-06-25 10:24:12 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349B68D70>
2026-06-25 10:24:12 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D349B69C70>
2026-06-25 10:24:12 | INFO | Database connections established
2026-06-25 10:24:12 | INFO | Collecting MIDs for: 2026-06-24
2026-06-25 10:24:13 | INFO | Found 868 MIDs
2026-06-25 10:24:13 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 10:24:13 | INFO | ================================================================================
2026-06-25 10:24:13 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-25 10:24:13 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-25 10:24:13 | ERROR | Failed processing table OQaD
Traceback (most recent call last):
File "D:\z\airflow-project\elt2.py", line 172, in elt
df=registry[f"fetch_{table_name}"]()(sql_engine, table_name , table_type, mids, run_date)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
TypeError: fetch_OQaD() missing 5 required positional arguments: 'engine', 'table_name', 'table_type', 'empids', and 'run_date'
2026-06-25 11:19:10 | INFO | Pipeline Start Date: 2026-06-24
2026-06-25 11:19:10 | INFO | ================================================================================
2026-06-25 11:19:10 | INFO | Hello from data-move Python data pipeline!
2026-06-25 11:19:10 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 11:19:10 | INFO | Connecting to databases...
2026-06-25 11:19:11 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002B17CBD0890>
2026-06-25 11:19:13 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002B17DCE0910>
2026-06-25 11:19:14 | INFO | Database connections established
2026-06-25 11:19:14 | INFO | Collecting MIDs for: 2026-06-24
2026-06-25 11:19:14 | INFO | Found 868 MIDs
2026-06-25 11:19:15 | INFO | Pipeline Run Date: 2026-06-24
2026-06-25 11:19:15 | INFO | ================================================================================
2026-06-25 11:19:15 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
2026-06-25 11:19:15 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-25 11:19:15 | INFO | Exists: True
2026-06-25 11:19:15 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
2026-06-25 11:19:15 | INFO | Fetching data for 868 EMPIDs
2026-06-25 11:19:15 | INFO | Fetching OQaD data for run_date=2026-06-24
2026-06-25 11:19:21 | INFO | fn name is fetch_OQad ------Fetched 469 rows
2026-06-25 11:19:21 | INFO | Fetched total row -: 469 from sql server for table-:OQaD ...........!!!
2026-06-25 11:19:21 | INFO | Fetched 469 rows
2026-06-25 11:19:21 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
2026-06-25 11:19:21 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-25 11:19:21 | INFO | Inserted rows 0 to 469
2026-06-25 11:19:21 | INFO | OQaD: inserted 469 rows into ClickHouse
2026-06-25 11:19:21 | INFO | OQaD loaded successfully (469 rows)
2026-06-25 11:19:21 | INFO | ================================================================================
2026-06-25 11:19:21 | INFO | Pipeline Completed Successfully
2026-06-25 11:19:21 | INFO | ================================================================================
2026-06-25 11:19:21 | INFO | Pipeline completed successfully. pipeline_trigeered_on_date=2026-06-25last_successful_run_date=2026-06-24
Reference in New Issue View Git Blame Copy Permalink