Feature/retrieval optimization

[farshad68]

Hey, I've resolved the merge conflicts between the optimization and the retrieval_optimization branch. Could you review the changes and verify your feature functionality remains intact?

Here is a summary of how the conflicts were handled:

Pipeline Execution (Main): I kept your old_loaders.json_loader ingestion logic, but I rejected the commented-out search and generation blocks. The active asyncio search and generation logic from the baseline is retained.

Search Logging & Dependency Injection (search.py):

I kept your profilingLogger additions and the user_config/opt_config dependency injections in search_authorized (critical for the optimizer).

Ingestion ETA (ingestion.py): I completely accepted your progress bar, batch tracking, and perf_counter logic.

Dataset Loading (tools.py): I rejected the custom pandas CSVDataset wrapper because it hardcoded paths. Instead, I kept the native HuggingFace datasets.load_dataset logic, but I added your sep=';' argument to ensure your specific European CSV formats still parse correctly.

SMAC Optimizer Fix (retrieval_optimization.py): Your optimization script was crashing with a RuntimeError because the tight ForbiddenAndConjunction rules caused SMAC to discard all random initial configurations. I added an initial_design injection to force SMAC to evaluate the default config so it can always start successfully.

Please pull the latest branch, run a test optimization, and let me know if the logging or CSV ingestion behaves as you expect.