Ankit Malik
241 lines
16 KiB
Plaintext
241 lines
16 KiB
Plaintext
2026-06-24 15:55:46 | INFO | Pipeline Start Date: 2026-06-23
|
|
2026-06-24 15:55:46 | INFO | ================================================================================
|
|
2026-06-24 15:55:46 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 15:55:46 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 15:55:46 | INFO | Connecting to databases...
|
|
2026-06-24 15:55:48 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8151D4730>
|
|
2026-06-24 15:55:52 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8162F0B90>
|
|
2026-06-24 15:55:52 | INFO | Database connections established
|
|
2026-06-24 15:55:52 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 15:55:52 | INFO | Found 863 MIDs
|
|
2026-06-24 15:55:52 | WARNING | Pipeline failed. Retry 1/3. Error: 'tables'
|
|
2026-06-24 15:55:57 | INFO | ================================================================================
|
|
2026-06-24 15:55:57 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 15:55:57 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 15:55:57 | INFO | Connecting to databases...
|
|
2026-06-24 15:55:58 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C81600B550>
|
|
2026-06-24 15:55:58 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8162F5A90>
|
|
2026-06-24 15:55:59 | INFO | Database connections established
|
|
2026-06-24 15:55:59 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 15:55:59 | INFO | Found 863 MIDs
|
|
2026-06-24 15:55:59 | WARNING | Pipeline failed. Retry 2/3. Error: 'tables'
|
|
2026-06-24 15:56:04 | INFO | ================================================================================
|
|
2026-06-24 15:56:04 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 15:56:04 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 15:56:04 | INFO | Connecting to databases...
|
|
2026-06-24 15:56:04 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8162F59A0>
|
|
2026-06-24 15:56:05 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C8162F68A0>
|
|
2026-06-24 15:56:05 | INFO | Database connections established
|
|
2026-06-24 15:56:05 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 15:56:05 | INFO | Found 863 MIDs
|
|
2026-06-24 16:10:30 | INFO | Pipeline Start Date: 2026-06-23
|
|
2026-06-24 16:10:30 | INFO | ================================================================================
|
|
2026-06-24 16:10:30 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 16:10:30 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:10:30 | INFO | Connecting to databases...
|
|
2026-06-24 16:10:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29EE8C470>
|
|
2026-06-24 16:10:32 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FEBCA50>
|
|
2026-06-24 16:10:32 | INFO | Database connections established
|
|
2026-06-24 16:10:32 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 16:10:33 | INFO | Found 863 MIDs
|
|
2026-06-24 16:10:33 | WARNING | Pipeline failed. Retry 1/3. Error: 'tables'
|
|
2026-06-24 16:10:38 | INFO | ================================================================================
|
|
2026-06-24 16:10:38 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 16:10:38 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:10:38 | INFO | Connecting to databases...
|
|
2026-06-24 16:10:38 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FBB7550>
|
|
2026-06-24 16:10:39 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FECD9A0>
|
|
2026-06-24 16:10:40 | INFO | Database connections established
|
|
2026-06-24 16:10:40 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 16:10:40 | INFO | Found 863 MIDs
|
|
2026-06-24 16:10:40 | WARNING | Pipeline failed. Retry 2/3. Error: 'tables'
|
|
2026-06-24 16:10:45 | INFO | ================================================================================
|
|
2026-06-24 16:10:45 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 16:10:45 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:10:45 | INFO | Connecting to databases...
|
|
2026-06-24 16:10:45 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FECD8B0>
|
|
2026-06-24 16:10:46 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B29FECE7B0>
|
|
2026-06-24 16:10:46 | INFO | Database connections established
|
|
2026-06-24 16:10:46 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 16:10:47 | INFO | Found 863 MIDs
|
|
2026-06-24 16:12:02 | INFO | Pipeline Start Date: 2026-06-23
|
|
2026-06-24 16:12:02 | INFO | ================================================================================
|
|
2026-06-24 16:12:02 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 16:12:02 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:12:02 | INFO | Connecting to databases...
|
|
2026-06-24 16:12:02 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DBAE05D0>
|
|
2026-06-24 16:12:03 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB8CA50>
|
|
2026-06-24 16:12:04 | INFO | Database connections established
|
|
2026-06-24 16:12:04 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 16:12:04 | INFO | Found 863 MIDs
|
|
2026-06-24 16:12:05 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:12:05 | INFO | ================================================================================
|
|
2026-06-24 16:12:05 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-24 16:12:05 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-24 16:12:05 | INFO | Exists: True
|
|
2026-06-24 16:12:05 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-24 16:12:05 | INFO | Fetching data for 863 EMPIDs
|
|
2026-06-24 16:12:05 | INFO | Fetching OQaD data for run_date=2026-06-23
|
|
2026-06-24 16:12:11 | INFO | fn name is fetch_OQad ------Fetched 458 rows
|
|
2026-06-24 16:12:11 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
|
|
2026-06-24 16:12:11 | INFO | Fetched 458 rows
|
|
2026-06-24 16:12:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-24 16:12:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-24 16:12:11 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "D:\z\airflow-project\elt.py", line 243, in elt
|
|
load_to_clickhouse(
|
|
~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
table_name=table_name,
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
df=df,
|
|
^^^^^^
|
|
)
|
|
^
|
|
File "D:\z\airflow-project\clickhouse_task\load_table.py", line 28, in load_to_clickhouse
|
|
chunk_df = df.iloc[start:end]
|
|
^^^^^^^
|
|
AttributeError: 'DataFrame' object has no attribute 'iloc'
|
|
2026-06-24 16:12:11 | WARNING | Pipeline failed. Retry 1/3. Error: 'DataFrame' object has no attribute 'iloc'
|
|
2026-06-24 16:12:16 | INFO | ================================================================================
|
|
2026-06-24 16:12:16 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 16:12:16 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:12:16 | INFO | Connecting to databases...
|
|
2026-06-24 16:12:17 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB6C650>
|
|
2026-06-24 16:12:17 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB99A90>
|
|
2026-06-24 16:12:18 | INFO | Database connections established
|
|
2026-06-24 16:12:18 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 16:12:18 | INFO | Found 863 MIDs
|
|
2026-06-24 16:12:18 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:12:18 | INFO | ================================================================================
|
|
2026-06-24 16:12:18 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-24 16:12:18 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-24 16:12:18 | INFO | Exists: True
|
|
2026-06-24 16:12:18 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-24 16:12:18 | INFO | Fetching data for 863 EMPIDs
|
|
2026-06-24 16:12:18 | INFO | Fetching OQaD data for run_date=2026-06-23
|
|
2026-06-24 16:12:24 | INFO | fn name is fetch_OQad ------Fetched 458 rows
|
|
2026-06-24 16:12:24 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
|
|
2026-06-24 16:12:24 | INFO | Fetched 458 rows
|
|
2026-06-24 16:12:24 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-24 16:12:24 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-24 16:12:24 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "D:\z\airflow-project\elt.py", line 243, in elt
|
|
load_to_clickhouse(
|
|
~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
table_name=table_name,
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
df=df,
|
|
^^^^^^
|
|
)
|
|
^
|
|
File "D:\z\airflow-project\clickhouse_task\load_table.py", line 28, in load_to_clickhouse
|
|
chunk_df = df.iloc[start:end]
|
|
^^^^^^^
|
|
AttributeError: 'DataFrame' object has no attribute 'iloc'
|
|
2026-06-24 16:12:24 | WARNING | Pipeline failed. Retry 2/3. Error: 'DataFrame' object has no attribute 'iloc'
|
|
2026-06-24 16:12:29 | INFO | ================================================================================
|
|
2026-06-24 16:12:29 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 16:12:29 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:12:29 | INFO | Connecting to databases...
|
|
2026-06-24 16:12:30 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB9A120>
|
|
2026-06-24 16:12:30 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000258DCB9A8A0>
|
|
2026-06-24 16:12:31 | INFO | Database connections established
|
|
2026-06-24 16:12:31 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 16:12:31 | INFO | Found 863 MIDs
|
|
2026-06-24 16:12:31 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:12:31 | INFO | ================================================================================
|
|
2026-06-24 16:12:31 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-24 16:12:31 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-24 16:12:31 | INFO | Exists: True
|
|
2026-06-24 16:12:31 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-24 16:12:31 | INFO | Fetching data for 863 EMPIDs
|
|
2026-06-24 16:12:31 | INFO | Fetching OQaD data for run_date=2026-06-23
|
|
2026-06-24 16:12:43 | INFO | fn name is fetch_OQad ------Fetched 458 rows
|
|
2026-06-24 16:12:43 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
|
|
2026-06-24 16:12:43 | INFO | Fetched 458 rows
|
|
2026-06-24 16:12:43 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-24 16:12:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-24 16:12:43 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "D:\z\airflow-project\elt.py", line 243, in elt
|
|
load_to_clickhouse(
|
|
~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
table_name=table_name,
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
df=df,
|
|
^^^^^^
|
|
)
|
|
^
|
|
File "D:\z\airflow-project\clickhouse_task\load_table.py", line 28, in load_to_clickhouse
|
|
chunk_df = df.iloc[start:end]
|
|
^^^^^^^
|
|
AttributeError: 'DataFrame' object has no attribute 'iloc'
|
|
2026-06-24 16:17:01 | INFO | Pipeline Start Date: 2026-06-23
|
|
2026-06-24 16:17:01 | INFO | ================================================================================
|
|
2026-06-24 16:17:01 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 16:17:01 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:17:01 | INFO | Connecting to databases...
|
|
2026-06-24 16:17:02 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000018D3B1F05D0>
|
|
2026-06-24 16:17:03 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000018D3C29CA50>
|
|
2026-06-24 16:17:04 | INFO | Database connections established
|
|
2026-06-24 16:17:04 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 16:17:04 | INFO | Found 863 MIDs
|
|
2026-06-24 16:17:04 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:17:04 | INFO | ================================================================================
|
|
2026-06-24 16:17:04 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-24 16:17:04 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-24 16:17:04 | INFO | Exists: True
|
|
2026-06-24 16:17:04 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-24 16:17:04 | INFO | Fetching data for 863 EMPIDs
|
|
2026-06-24 16:17:04 | INFO | Fetching OQaD data for run_date=2026-06-23
|
|
2026-06-24 16:17:10 | INFO | fn name is fetch_OQad ------Fetched 458 rows
|
|
2026-06-24 16:17:10 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
|
|
2026-06-24 16:17:10 | INFO | Fetched 458 rows
|
|
2026-06-24 16:17:10 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-24 16:17:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-24 16:17:11 | INFO | Inserted rows 0 to 458
|
|
2026-06-24 16:17:11 | INFO | OQaD: inserted 458 rows into ClickHouse
|
|
2026-06-24 16:17:11 | INFO | OQaD loaded successfully (458 rows)
|
|
2026-06-24 16:17:11 | INFO | ================================================================================
|
|
2026-06-24 16:17:11 | INFO | Pipeline Completed Successfully
|
|
2026-06-24 16:17:11 | INFO | ================================================================================
|
|
2026-06-24 16:17:11 | INFO | Pipeline completed successfully. pipeline_trigeered_on_date=2026-06-24last_successful_run_date=2026-06-23
|
|
2026-06-24 16:17:31 | INFO | Pipeline Start Date: 2026-06-23
|
|
2026-06-24 16:17:31 | INFO | ================================================================================
|
|
2026-06-24 16:17:31 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-24 16:17:31 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:17:31 | INFO | Connecting to databases...
|
|
2026-06-24 16:17:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000022F667F05D0>
|
|
2026-06-24 16:17:33 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000022F678CCA50>
|
|
2026-06-24 16:17:33 | INFO | Database connections established
|
|
2026-06-24 16:17:33 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-24 16:17:34 | INFO | Found 863 MIDs
|
|
2026-06-24 16:17:34 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-24 16:17:34 | INFO | ================================================================================
|
|
2026-06-24 16:17:34 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-24 16:17:34 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-24 16:17:34 | INFO | Exists: True
|
|
2026-06-24 16:17:34 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-24 16:17:34 | INFO | Fetching data for 863 EMPIDs
|
|
2026-06-24 16:17:34 | INFO | Fetching OQaD data for run_date=2026-06-23
|
|
2026-06-24 16:17:40 | INFO | fn name is fetch_OQad ------Fetched 458 rows
|
|
2026-06-24 16:17:40 | INFO | Fetched total row -: 458 from sql server for table-:OQaD ...........!!!
|
|
2026-06-24 16:17:40 | INFO | Fetched 458 rows
|
|
2026-06-24 16:17:40 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-24 16:17:41 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-24 16:17:41 | INFO | Inserted rows 0 to 458
|
|
2026-06-24 16:17:41 | INFO | OQaD: inserted 458 rows into ClickHouse
|
|
2026-06-24 16:17:41 | INFO | OQaD loaded successfully (458 rows)
|
|
2026-06-24 16:17:41 | INFO | ================================================================================
|
|
2026-06-24 16:17:41 | INFO | Pipeline Completed Successfully
|
|
2026-06-24 16:17:41 | INFO | ================================================================================
|
|
2026-06-24 16:17:41 | INFO | Pipeline completed successfully. pipeline_trigeered_on_date=2026-06-24last_successful_run_date=2026-06-23
|