Challenges with Integrating External LLM Services

Limited Control Over Timeouts:
External LLM services have factors beyond our control that affect response times, such as server status fluctuations.

Variable LLM Processing Time:
LLMs inherently have unpredictable processing times, especially for complex tasks beyond simple responses, like searching for information in a file.

Other Factors Increasing Complexity:
Uploading data and managing its times, file size limitations, and evolving LLM APIs with potentially limited documentation add complexity to integration.

Mitigation Strategies:

Generous Timeouts: 
Account for external factors by implementing more lenient timeouts compared to internal and more “classical” services.

Error Handling:
Carefully monitor and address the various ever changing error messages that may arise during development and testing.


LLM-Specific Challenges:

Data Interaction Issues:
LLMs may encounter errors when reading data files due to their non-deterministic nature (unpredictable steps followed when interacting with a file)

Potential for Hallucination:
LLMs can generate seemingly plausible but inaccurate information ("hallucination") which can be misleading if you do not have a clear idea about the data being consulted.