Setting Up Autogen with a Local LLM Using LM Studio

Discover how to utilize the local Inference Server functionality of LM Studio to leverage an open-source local LLM with Autogen. This cost-efficient solution is a great alternative to OpenAI models, especially when dealing with extensive token usage.

Downloading and Installing LM Studio

Begin by downloading LM Studio from lmstudio.ai for your operating system. LM Studio supports Mac, Windows, and is in beta for Linux. This tutorial uses the Windows version.

Introduction to Large Language Models (LLMs)

If you’re new to LLMs, consider watching Andrej Karpathy’s “Intro to Large Language Models” to understand their training and significance.

Selecting an LLM

Choose from various open-source LLMs like Llama-2-7B-chat-GGUF or Mistral-7B-Instruct-v0.1-GGUF. Huggingface’s leaderboard offers a comparison based on parameters and specifications. Keep an eye out for new models on the platform.

Using LM Studio

Within LM Studio, search for LLMs based on your needs, like uncensored content or specific functionalities. The Home tab suggests models like Mistral 7B Instruct v0.1. Note that the download time varies based on model size.

Setting Up a Local Inference Server

After downloading the LLMs, navigate to the local inference server tab in LM Studio. Here, you can start the server and choose from the downloaded models. Keep note of the API path and port for later use.

Implementing Autogen

Create a new project folder and open it in Visual Studio Code. Set up a virtual environment, install pyautogen using pip, and create app.py. Import Autogen agents and configure the settings for either the paid or free solution.

Testing with OpenAI and Local LLM

Initiate a test by asking a question like “What is LLM Quantization?” Compare the responses from OpenAI’s paid service and the local free LLM like Mistral. Switch between paid and free solutions in the config list as needed.

Conclusion

Autogen can be run locally for free using open-source LLMs. While there are times when a paid solution is necessary, open-source options can be sufficient or even superior for specific tasks.

Remember, the method to run Autogen locally varies based on your device and OS. LM Studio offers a simple solution for Windows and Mac users. Stay tuned for future tutorials on other platforms like Ollama for Mac and Linux.

Good luck with running “autogen” for free!