Exploring Open-Source LLMs with LM Studio: A Step-by-Step Guide

While commercial large language models (LLMs) like OpenAI’s GPT-4 are excellent tools for learning modern AI and building agents, they come with certain drawbacks. These include costs, reduced data privacy, security concerns, and dependency on external services. 

As a result, the development of open-source LLMs has gained significant momentum, offering alternatives that are both accessible and customizable. Thanks to advancements in tools, running LLMs locally has become easier than ever. 

In this blog, we’ll explore how to use LM Studio, a user-friendly tool for downloading, hosting, and experimenting with open-source LLMs on your local machine.

How to Run Open Source LLMs with LM Studio?

You can explore different Open Source LLMs models locally on your machine by following these steps:

Step 1: Downloading and Installing LM Studio

Go to LM Studio Website https://lmstudio.ai/ and download the software for your operating system (Windows, Mac, or Linux).

Once the download is complete, follow the installation instructions for your OS. Note that some versions of LM Studio may still be in beta, so you might need to install additional tools or libraries depending on your system.

After installation, open the software. You’ll be greeted with the main home page, which displays a list of trending models and their specifications. This page is a great resource for understanding the capabilities and requirements of different models.

Step 2: Exploring and Downloading Models

Use the search bar to find models that suit your needs. You can filter models based on their size, token context, or intended use case (e.g., chat completions).

LM Studio has a handy feature that analyzes your hardware and estimates how well a specific model will run on your system. 

Click the Compatibility Guess button to see if your system can handle the model.

Once you’ve found a model that works for your system, click the download button. Feel free to experiment with a few if you’re unsure which model to choose. LM Studio makes it easy to explore and compare different models.

Step 3: Running a Model Locally

After downloading a model, go to the Chat page in LM Studio. Use the drop-down menu at the top of the page to select the model you want to load. A progress bar will indicate when the model is ready.

Enable GPU (Optional)
If your system has a GPU, you can enable it to speed up model inference. LM Studio allows you to allocate part or all of your GPU for this purpose. The performance status at the bottom of the page will show how the GPU affects response times.

Once the model is loaded, you can start typing prompts into the chat interface. Experiment with different prompts to see how well the model performs for your specific use case.

Step 4: Serving an LLM Locally

To run an LLM as a local server, navigate to the Server page in LM Studio.
Select the model you want to serve from the drop-down menu.

Click the Start Server button. Once the server is running, you’ll see examples of how to connect to it using code.

Connect to the Server
You can use the provided Python code examples to interact with the server. For instance, the following code connects to the server and sends a prompt:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")

completion = client.chat.completions.create(

  model="local-model",  # Model name (can be anything)

  messages=[

    {"role": "system", "content": "Always answer in rhymes."},

    {"role": "user", "content": "Introduce yourself."}

  ],

  temperature=0.7,

)

print(completion.choices[0].message)

Run this code in your preferred IDE (e.g., VS Code) to interact with the locally hosted model.

If you encounter issues, ensure that the Server Model Settings match the loaded model. For example, if you’re using a chat-based model, make sure the server settings are configured accordingly.

Step 5: Experimenting and Evaluating Models

Use the chat interface to test various prompts and evaluate how well the model performs. This is a great way to determine if a model is suitable for your specific use case.

For a more systematic approach to evaluating prompts and models, consider using tools like Prompt Flow.

Why Use LM Studio?

LM Studio is an excellent tool for anyone looking to explore open-source LLMs without relying on commercial services. It simplifies the process of downloading, running, and serving models locally, making it accessible even for beginners. 

By hosting models on your machine, you gain greater control over data privacy, reduce costs, and eliminate dependencies on external providers.

With LM Studio, experimenting with open-source LLMs has never been easier. Whether you’re a developer, researcher, or AI enthusiast, this tool provides a seamless way to explore the capabilities of various models, run them locally, and even serve them for custom applications. 

By following the steps outlined in this guide, you can start building and testing AI agents with confidence, all while maintaining full control over your data and resources. Happy experimenting!