However, if we are to be honest, the history of async usage in Django wasn't very impressive. It was always clunky, and you end up with your code cluttered with this ugly syntax.

async_to_sync(some_function)(arg_1, arg_2)

You could argue that for most products, you don’t really need async. It was just an extra layer of complexity without any significant practical benefit.

Over the last couple of years, AI use-cases have changed that perception. Many AI products have calling external APIs over the network as their bottleneck. This makes the complexity from async Python worth considering. FastAPI with its intuitive async usage and simplicity have risen to be the default API/web layer for AI projects.

We watched with concern as the view of Django as a clunky async framework spread. This happened partly because large language models had outdated information and partly because there aren't any complex or large open-source projects showcasing Django's async usage.

One of our side quests when building ColiVara was to demonstrate Django's async features and to be completely end-to-end async. We aimed to be an open-source project that could effectively showcase Django's async capabilities.

As background, ColiVara is a retrieval API that allows you to store, search, and retrieve documents based on their visual embeddings. It works exactly like RAG from the end-user standpoint - but using vision models instead of chunking and text-processing for documents.

It is designed as a Django API service with a separate, standalone GPU service that handles the AI workloads.

Footguns

The key takeaway is that your application needs full async support to truly benefit from it. It's a commitment you make from the start, and mixing sync and async code won't give you any benefits. It will only add unnecessary complexity.

If your code is a mix between sync and async, Django has to mimic the other call style to make your code work. This switch causes a slight performance delay of about a millisecond. Is it a big deal? Probably not if you have a sync call occasionally. However, if your application is a 50/50 mix, it's likely better to stick with sync code.

In summary, this means you must:

• Use an ASGI web server to handle requests asynchronously

• Write async views

• Use async queries in the ORM

• Use an async HTTP client library like aiohttp for any API calls

• Upgrade your custom middleware to be async-compatible

ASGI Web Server

For a Django project, the usual choices are either Daphne or Uvicorn. We think both have similar capabilities. We chose Uvicorn because it's lighter and more like "gunicorn." Our general rule is to use Daphne if we're working with WebSockets and Django Channels; otherwise, we go with Uvicorn.

Async views

We used django-ninja as our API library. It is newer than the well-known Django Rest Framework and focuses on performance and support for async code.

Our experience with django-ninja has been rock solid, and we recommend starting new API Django projects with it, even if you aren't writing async views or endpoints.

The coding style is very similar to FastAPI, but you also get all the Django features that make it "batteries-included."

Async ORM

The biggest improvement in Django over the last six months is how much the ORM now supports async code. Practically, the vast majority of operations are supported. You simply add a prefix of a and things work out of the box.

User.objects.get(id=1) # sync
User.objects.aget(id=1) # async

One thing to be mindful of, is all the LLMs as of late 2024 are completely outdated on what is possible and what is not possible with Django async ORM operations. You should assume that their code is wrong, especially if you see it littered with async_to_sync.

Async API calls

Here is where the big benefit for async paid off for us. The architecture of ColiVara is to call the GPU service whenever we need to do an AI workload. This is how this looks like:

# stuff here 
async with aiohttp.ClientSession() as session:
        async with session.post(
            EMBEDDINGS_URL, json=payload, headers=headers
        ) as response:
            if response.status != 200:
                #handle error
            response_data = await response.json()
# stuff here

In this code, when our application hits the await keyword, it can yield control back to the event loop, allowing other tasks to run while waiting for the HTTP response.

Here's what happens:

1. When the code reaches the POST request, it doesn't block

2. While waiting for the response from EMBEDDINGS_URL, the event loop can:

   • Handle other incoming requests

   • Process other async tasks

   • Run other coroutines

3. When the response comes back, our code resumes from where it left off

This is one of the main benefits of async programming - it allows for concurrent execution without using multiple threads or processes, particularly efficient for I/O-bound operations like HTTP requests.

Additionally, we can easily process items concurrently which leads to a big performance boost.

async def process_document(document):
    # stuff here
    return 

documents = ['A', 'B', 'C', 'D']
# Process all items concurrently
results = await asyncio.gather(
        *[process_document(document) for document in documents]
    )

Async middleware

One of the main challenges with async Django is middleware. If you use synchronous middleware between an ASGI server and an async view, it switches to sync mode for the middleware and back to async for the view, causing overhead. Designing middleware to work both async and sync is simple; just be mindful of it. Here's a basic middleware that adds a slash to API requests as an example.

from asgiref.sync import iscoroutinefunction
from django.utils.decorators import sync_and_async_middleware


# add slash middleware
@sync_and_async_middleware
def add_slash(get_response):
    if iscoroutinefunction(get_response):
        async def middleware(request):
            # we want to leave openapi, swagger and redoc as is
            keep_as_is = any(
                x in request.path for x in ["openapi", "swagger", "redoc", "docs"]
            )
            if not request.path.endswith("/") and not keep_as_is:
                request.path_info = request.path = f"{request.path}/"
            return await get_response(request)
    else:
        def middleware(request):
            keep_as_is = any(
                x in request.path for x in ["openapi", "swagger", "redoc", "docs"]
            )
            if not request.path.endswith("/") and not keep_as_is:
                request.path_info = request.path = f"{request.path}/"
            return get_response(request)

    return middleware

Django will automatically pick the sync vs. the async path depending on the request.

We recommend thoroughly reviewing the middleware you use for async Django to ensure they support async usage. The switch overhead is generally minimal, but it could become problematic if left unchecked.

Conclusion

We believe async Django is ready for production. In theory, there should be no performance loss when using async Django instead of FastAPI for the same tasks. Django's built-in features greatly simplify and enhance the developer experience. Using async involves some complexity, as the entire code path must be async. In AI workloads, this complexity is often a worthwhile tradeoff.

A user at some point later asks a question that get turned also into a numerical representation. We compare the numbers and do some math to identify the top k(3) chunks of texts that matches the question. This is feed that into an LLM, and we get an answer based on the documents uploaded.

RAG implementations are notorious for being unscientific in nature, and more art than science. How do you transform a PDF into text? What about tabular data? What if there are pictures in the document and they are pretty important? How long should the chunks be? How many chunks? Should you use Cosine similarity for the math or something else? and a million other questions where the correct answers is always "it depends".

I am and you should be highly suspicious of generic "one size fits all" RAG solutions. The really good ones customize for the use-case, user types, and the documents.

But, that's all irrelevant now, I am moving the project to use long-context instead because I think no one should be building RAG products in 2024 - and I will explain why below.

RAG is a workaround

In 2021/2022 - when I first started building on LLMs we had GPT3.5 and ~4000 context length. LLMs without guardrails hallucinated at unacceptable rate for anything that is accuracy-critical.

Adding a knowledge base solved hallucination to a large extent. But, you were only limited to 4000 words. Many real-world problems required documents that are larger than the model context. So, RAG become the standard solutions.

Break down large documents into smaller pieces and use semantic search techniques to get the relevant parts, instead of the whole documents. A clever solution to the problem of context length and hallucination.

Now - we have neither of these problems. Gemini Pro handles 1m+ tokens with no issues, and and LLMs hallucinate a lot less. Some have seen degradation in GPT-4 after 16k tokens, but this more of a GPT-4 limitations than all LLMs.

This is my favorite formal eval (link) for the "real context size" for LLMs and we can clearly sees that context degradation is model-specific and not an LLM limitation.

In summary - we used RAG because we had a problem that no longer exists.

RAG is less performant than long-context

As of late July 2024, we had pretty solid evidence and evals that long-context beats most RAG implementations. I like this paper as it proves this point, even though it tries to tell you to use some RAG.

If you care about performance, then you absolutely should take away RAG from your stack.

KISS (Keep it simple) always wins

Some people argue that inference cost is a factor, and you should consider RAG for some queries (like the paper above). Nope. Bad idea.

1. LLM inference costs are coming down rapidly

2. The development cost (AKA all the engineers you need to hire to maintain your bespoke RAG solutions) is a lot more expensive than the API costs

3. Large RAG applications are very difficult to maintain and iterate on, because they are fragile and small changes can lead to big changes.

Long-context applications on the other hand are very simple. You just worry about getting the text out of documents correctly and everything else works via calling an API. No chunking, no vectors, no math. Easy to maintain, easy to iterate on, and as simple as it gets.

The unpopular opinion

Simplicity is great, but it doesn't sell. So, there would be a lot of noise and disagreement to folks invested in the RAG ecosystem. I get it. If you just raised a bunch of money for a vector DB, you aren't a big fan of this new development.

I ran a large RAG application in production and have been for the last 2.5 years. My skill-set and experience in that corner of development are essentially obsolete. But, it is the nature of AI engineering and being on the bleeding edge.

I wanted them to use screenshot-to-code to whip up code templates directly, instead of just handing over Figma designs. They're savvy enough to tweak Tailwind classes and work with AI prompts. But let's be real, setting up a GitHub repository, messing with Docker, and managing Python environments? That’s a bit much to ask.

One problem with hosting AI tools in the cloud is managing access. You have to think about security and who can do what. After all, those AI API calls aren't free, and we can't just let everyone in.

I also needed a simple, quick method to manage who gets in and who doesn’t. And with a bunch of other tools to host, the last thing I wanted was to drown in a massive, 200-line Nginx config file.

That’s where Cloudflare Zero-Trust swoops in to save the day!

Tools and Concepts:

When you're gearing up to host an open source project in the cloud, there are a handful of tools and concepts that really come in handy. I'm going to walk you through some of the essentials. Keep in mind, my recommendations are based on personal experience, so they might be a bit subjective.

1. Cloud Server: getting a computer from a cloud provider

2. SSH and Vs Code: Accessing this computer from vs code

3. Setting up project: Setup the project on the cloud server

4. Cloudflare Site setup: Setting up your site in cloudflare

5. Cloudflare Tunnel: Getting your server to tunnel traffic to Cloudflare

6. Cloudflare Access Control: Setting up access rules

7. Recipe: Step by step directions.

If you're already familiar with these concepts, feel free to jump straight to the Recipe section where you can follow the practical steps to get your project live.

Cloud server

Alright, let's dive into the cloud server setup. Once you get a taste of hosting your own tools, I bet you'll be hooked and start self-hosting everything! The first move? Grab yourself a server from a cloud provider.

I'm a fan of Hetzner, but honestly, any provider will do the trick and the setup steps are pretty similar. You just need a virtual private server (VPS). A computer in the cloud so to speak.

First things first, head over to accounts.hetzner.com/signUp to open your Hetzner account.

Now, if you’re tuning in from the U.S., brace yourself for a bit of a song and dance—it’s almost like signing up for a mortgage! Just kidding, but you will need to pay invoices in advance or setup direct bank transfers.

Once you're in, navigate to Hetzner Cloud from your dashboard, kick off a new project, and get ready to add a server. Here’s how I set mine up:

• Location: Pick the closest one to you. For me, that was Ashburn, VA, in the US East.

• Image: You can go with Ubuntu OS or, if you’re planning to use Docker, jump over to Apps and pick Docker CE for the latest Ubuntu with Docker Community Edition already installed. Of course, you can always install Docker on your own later.

• Type: I went with a Shared cVPU x86 (Intel/AMD) CPX11 - sporting 2GB RAM and a 40GB SSD.

• Networking: Opt for both IPv4 and IPv6. Skipping IPv4 to save a penny might sound tempting, but trust me, you’ll need it.

• SSH Keys: Definitely add an SSH key. If you haven’t done this before, Hetzner has a solid guide on generating SSH keys. This will allow you go into the server from anywhere. So, make sure to keep it secure and in accessible location. You will need the path to private key in the next step.

For now, you can skip setting up volumes, firewalls, backups, placement groups, labels, and cloud configs. You can always circle back to those later if you need them.

Give your serve a name and then hit ‘Create & Buy Now’ and you're all set to start your self-hosting adventure!

SSH

I won't go to all the intricacies of SSH here. Our goal is simple. Hit a button, and be able to jump into your cloud server with VS code all setup for you there. Our ideal world is you can't tell the difference between your local computer, and the one in the cloud. Here’s how you get it all set up:

1. Install the SSH Extension: First up, you need the right tool for the job. Open VS Code, head over to the Extensions view by clicking on the square icon on the sidebar. Search for "Remote-SSH" and install it. This extension is your golden ticket to connecting directly to your remote server.

2. Create or Edit the SSH Config File: You need an SSH configuration file to connect via hitting a button. In your VS code, open Command Palette Ctrl+Shift+P or on Mac command+shift+P . Choose Remote SSH: Open SSH Configuration File

   

3. Add Your Server Details: Open this config file in any text editor, and add a block for your server like this:

    Host my-server
        HostName your-server-ip 
        User your-username
        IdentityFile ~/path/to/your/private/key
   

   Host is whatever you decide to call your server. HostName is the server public IP address - available from the Hetzner dashboard. The user is typically root for Ubuntu servers. The IdentityFile is where your SSH private key is saved on your local machine.

   

4. Connect Using VS Code: Now, open VS Code, launch the Command Palette (Ctrl+Shift+P), and type ‘SSH: Connect to Host’. Instead of entering all the details, you’ll just see ‘my-server’ (or whatever nickname you gave your server) listed there. Click it, and VS Code will use the details from your config file to log you in automatically—no hassle required.

5. Enjoy Seamless Access: And there you have it! With your SSH config file in place, connecting to your remote server becomes a breeze. Just a couple of clicks, and you’re in, ready to get down to business. Whatever you can do in your local machine, you can do in the remote server.

Setting up the project

This is a pretty straightforward and easy step with one simple rule. Whatever you do in your local machine to setup an open-source project. Do it in the remote server after you SSH.

For all intents and purposes, it is just another computer and works the same way. For example, setting up screenshot-to-code. Here’s how you’d kick things off:

First up, make sure you've got Docker, Docker Compose, and Git on your server. Sometimes, depending on your cloud provider and the server setup you chose, these might already be installed. If not, no sweat—the Docker and Git websites have super straightforward instructions on how to get them up and running.

1. Git clone the package.

    git clone https://github.com/abi/screenshot-to-code.git
   

2. Spin the docker image up.

    echo "OPENAI_API_KEY=sk-your-key" > .env
    docker-compose up -d --build
   

And just like that, if you head over to <server-ip>:5173 in your web browser, you’ll see your application live and in action.

Congratulations! You’ve just successfully hosted your first open-source application. Pretty cool, right?

Here’s the good news. You can access this tool from anywhere in the world, and so can your team.

And the not-so-good news? Well, technically, anyone in the world could access it too. So, let's make sure we handle that access wisely in the next steps!

Cloudflare Site Setup

Alright, let’s get your application locked down with Cloudflare! Managing who gets to access your app is super important, and for that, you'll need to add a new site to Cloudflare. Plus, having a domain on Cloudflare not only ticks off a technical requirement but also keeps things neat—because let’s face it, memorizing IP addresses isn’t fun.

First, you will need a cloudflare account, you can sign up here: https://dash.cloudflare.com/sign-up

Don’t have a domain name yet? No worries—you can snag one directly through Cloudflare Registrar. If you’ve already got a domain, perfect! You can use it here without any hassle.

Now, the steps to add a new site vary a bit depending on your domain registrar, but here’s a link to Cloudflare’s detailed documentation on how to do add a new site. The exact steps depends on who is your domain registrar. The free tier should be just fine for this project.

When your done, your dashboard should look like this:

Cloudflare Tunnel

Here is the overall idea behind cloudflare tunnel in a simple way. You download something called "cloudflared" on your server. This opens up a connection to Cloudflare servers - a "tunnel". Once the tunnel is established, Cloudflare acts as a middleman. It securely routes traffic from the internet through the tunnel to your server, down to the port number.

Here is the steps to get it.

1. Configure Zero Trust for your Cloudflare Account. Again, the free tier is fine for our use-case

   

   2. Go to Networks > Tunnels from the Zero Trust dashboard after the initial setup

   3. Select Create a tunnel.

   4. Choose Cloudflared for the connector type and select Next.

   5. Enter a name for your tunnel. This is a tunnel to your server, so make sure the name reflects that.

   6. Select Save tunnel.

   7. Next, you will need to install cloudflared and run it in your remote server.

   8. SSH into the remote machine as described before.

   9. If you are on Hetzner and used a Ubuntu server, copy and paste this command. You can find that command in the Tunnel dashboard for different type of operating systems. It will take a few mins until your tunnel is active.

    curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb && 

    sudo dpkg -i cloudflared.deb && 

    sudo cloudflared service install <token from Zero trust tunnel setup>

10. Once the command has finished running, your connector will appear in Zero Trust. Select Next to continue to the next step.

11. In the Public Hostnames tab, choose the Domain that you setup before. I recommend specifying a subdomain for each open-source project you are hosting. For example: screenshot-to-code.your_domain.com. That way you can use the same server and domain for 100's of projects.

12. Specify a service, for example, the screenshot-to-code project uses port 5173. So, it would be http://localhost:5173.

    1. If you have another project on port:8001, then you just create another public hostname. new-project.your_domain.com and the service would be http://localhost:8001 - no need for a new setup or servers!

13. Select Save tunnel.

And that's it. Now, if you navigate to screenshot-to-code.your_domain.com, you should be able to see your application. No Nginx or SSL setup needed. Cloudflare took care of all of these for us.

We still have a problem though. Our application is still wide-open to anyone who manages to access the domain name. This what Cloudflare Access Control fixes.

Cloudflare Access Control

So, our mission is pretty clear here: We only want the right people getting into our application. Here’s how we’ll make sure of that:

When someone heads over to our app at screenshot-to-code.your-domain.com, Cloudflare steps in and greets them with a login page. All they need to do is punch in their email.

Now, if their email is on our VIP list, they’ll get a PIN number sent right to it. Pop in that PIN, and voilà, they’re in! If not, no access—simple as that.

Down the road, we can jazz things up with all sorts of fancy authentication methods—think single sign-on, tokens, even hardware solutions, complete with custom rules to fit our needs. But for now, this email and PIN setup does the trick quite nicely.

Here is steps to do that.

1. Go to Access > Applications from the Zero Trust dashboard

2. Click on Add an Application button

3. Click on Self-hosted option

4. Give your application a name. For example, screenshot-to-code and how long the access sessions are good for.

5. Add your domain and subdomain

6. Keep everything else the same and move to policies.

7. Give your policy a name email-rule for example. And under Configure Rules, add Selector to be email and add the emails of the users you want to have access.

   

8. Click on the Authentication tab, and set the Authentication to be One Time Pin

   

9. Keep the default Settings without any changes.

10. Save your Applications

And that's it. Now, if you navigate to screenshot_to_code.your-domain.com you will be able to see Cloudflare default log in page. Only the people who you gave access to will be able to log in and access the application.

For everyone else - they can't access your application.

Recipe:

The goal of this recipe is self-host an open-source AI application on the cloud in a secure manner. Specifically to restrict access based access rules and policies. We are using the screenshot-to-code as an example.

1. Start by getting a VPS if you don't have one already. Make sure you can access it via SSH.

2. Make sure Docker, docker-compose and git are installed on your VPS

3. SSH into the machine and setup the screenshot-to-code project

    git clone https://github.com/abi/screenshot-to-code.git 
    echo "OPENAI_API_KEY=sk-your-key" > .env
    docker-compose up -d --build
   

4. Make sure project is working with no errors by navigating to <public_ip>:5173 in your browser or docker-compose logs in your terminal

5. Set up a new Cloudflare Site. You will need a cloudflare account and a domain name.

6. Setup Zero Trust for your Cloudflare Account. The free tier is fine for our use-case. You will need to add your credit card though.

7. Create a cloudflare tunnel on your Zero Trust dashboard

   1. Go to Networks > Tunnels from the Zero Trust dashboard after the initial setup

   2. Select Create a tunnel.

   3. Choose Cloudflared for the connector type and select Next.

   4. Enter a name for your tunnel. This is a tunnel to your server, so make sure the name reflects that.

   5. Select Save tunnel

8. Connect your VPS to the tunnel by installing cloudflared

   1. SSH into the remote machine

   2. If you are on a Ubuntu server, copy and paste this command. You can find that command in the Tunnel dashboard for different type of operating systems. It will take a few mins until your tunnel is active.

   3. Once the command has finished running, your connector will appear in Zero Trust. Select Next to continue to the next step

    curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb && 

    sudo dpkg -i cloudflared.deb && 

    sudo cloudflared service install <token from Zero trust tunnel setup>

9. Setup your application traffic to be sent via the tunnel

   1. In your Tunnel setup, under the Public Hostnames tab, choose a Domain that you setup before as part of setting a Cloudflare Website. Optionally, specify a subdomain for each open-source project you are hosting. For example: screenshot-to-code.your_domain.com. That way you can use the same server and domain for other projects.

   2. Specify a service, for example, the screenshot-to-code project uses port 5173. So, it would be http://localhost:5173.

   3. Select Save tunnel.

10. Lock your application behind Cloud Flare Access Control

    1. Go to Access > Applications from the Zero Trust dashboard

    2. Click on Add an Application button

    3. Click on Self-hosted option

    4. Give your application a name. For example, screenshot-to-code and how long the access sessions are good for.

    5. Add your domain and subdomain

    6. Keep everything else the same and move to policies.

    7. Setup an access policy by specifying a name email-rule for example. And under Configure Rules, add Selector to be email and add the emails of the users you want to have access.

    8. Click on the Authentication tab, and set the Authentication to be One Time Pin

    9. Keep the default Settings without any changes.

    10. Save your Applications

And that's it. Now, if you navigate to screenshot_to_code.your-domain.com you will be able to see Cloudflare default log in page. Only the people who you gave access to will be able to log in and access the application.

Congratulations, you have self-hosted your first open-source project with all what you need for future security concerns and scaling.

The idea is, give an LLM a query that is better answered via code execution instead of its training. Run the code in AgentRun, then return the answer to the user. It is more or less a proof of concept, that can be expanded on with additional tools that an LLM can use.

For this experiment, I had Ollama installed and running as well as the AgentRun API. My goal was use code generated by an LLM it to answer some questions that normally an LLM would struggle with. Like, what is 12345 * 54321? Or what is the largest prime number under 1000?

The full code is available here: https://jonathan-adly.github.io/AgentRun/examples/ollama_llama3/

Step 1: Setting Up

If you don't have Ollama installed, first install it from here. Then, run a test query to make sure everything is working.

curl -X POST http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt":"What is 1+1?"
 }'

Next, install AgentRun and have its REST API running. You will need docker installed to use docker-compose.

git clone https://github.com/Jonathan-Adly/agentrun
cd agentrun/agentrun-api
cp .env.example .env.dev
docker-compose up -d --build

And again, let's make a test request to make sure everything is running correctly.

curl -X GET http://localhost:8000/v1/health/
# {"status":"ok"}

Next, we will run a Python script that will be our starting point to run queries against Llama3 with Agentrun.

python -m venv agentrun-venv
# windows: .\agentrun-venv\Scripts\activate
source agentrun-venv/bin/activate
# windows: New-Item main.py -type file
touch main.py

pip install requests json_repair

In the file, we will start off by importing the necessary libraries. We'll need json for handling data and requests for making HTTP calls. We’re also using a cool library called json_repair just in case our JSON data decides to act up and we need to fix it on the fly. This is especially the case if use 8B version of Llama3 where the JSON sometimes is slightly broken.

import json
import json_repair
import requests

Step 2: Define the Function & Tools

We've crafted a simple function execute_python_code. This function is pretty straightforward—it sends a Python code snippet to a code execution environment provided by AgentRun and fetches the output.

Here's a quick peek at how this works:

def execute_python_code(code: str) -> str:
    code = json.dumps({"code": code})
    response = requests.post(
        "http://localhost:8000/v1/run/",
        data=code,
        headers={"Content-Type": "application/json"},
    )
    print(code)    
    output = response.json()["output"]
    return output

We basically format the code snippet into JSON, send it off to our localhost where the magic happens, and get back the result. You can read more about how AgentRun works here.

Next, we would use this function as our basis for defining the tool that we want Llama3 to use. Here is what this looks like.

tools = [
    {
        "type": "function",
        "function": {
            "name": "execute_python_code",
            "description": """Sends a python code snippet to the code execution environment and returns the output. 
            The code execution environment can automatically import any library or package by importing. 
            The code snippet to execute must be a valid python code and must use print() to output the result.""",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {
                        "type": "string",
                        "description": "The code snippet to execute. Must be a valid python code. Must use print() to output the result.",
                    },
                },
                "required": ["code"],
            },
        },
    },
]

Lastly, we will set up our model here. We can use the base Llama3 or any of the finetunes provided by the community. For the sake of experimentations, I ran my experiment using Dolphin-llama3 8b finetune.

# Ollama dolphin-llama3 page: https://ollama.com/library/dolphin-llama3
MODEL = "dolphin-llama3"

Step 3: The Integration with Ollama and Llama3

Moving on to the cooler element—integration with the Ollama and Llama3.

Here’s a how the query processing and tool selection works:

def generate_full_completion(prompt: str, model: str = MODEL) -> dict[str, str]:
    # setting up the parameters including our model
    params = {
        "model": model,
        "prompt": prompt,
        "stream": False,
        # seed and temperature for deterministic output
        "temperature": 0,
        "seed": 123,
        # format is JSON, since we are interested in tools/function calling
        "format": "json",
    }
    # making the post request and handling responses
    try:
        response = requests.post(
            f"http://localhost:11434/api/generate",
            headers={"Content-Type": "application/json"},
            data=json.dumps(params),
            timeout=60,
        )
        return json_repair.loads(response.text)
    except requests.RequestException as err:
        return {"error": f"API call error: {str(err)}"}

Step 4: Putting It All Together

Now, that we have everything setup. We will simply use a prompt to nudge the model toward using our execute_python_code tool for its outputs.

def get_answer(query: str) -> str:
    functions_prompt = f"""
        You have access to the following tools:
            {tools}
        You must follow these instructions:
        If a user query requires a tool, you must select the appropriate tool from the list of tools provided.
        Always select one or more of the above tools based on the user query
        If a tool is found, you must respond in the JSON format matching the following schema:
        {{
        "tools": {{
            "tool": "<name of the selected tool>",
            "tool_input": <parameters for the selected tool, matching the tool's JSON schema
        }}
        }}
        If there are multiple tools required, make sure a list of tools are returned in a JSON array.
        If there is no tool that match the user request, you will respond with empty json.
        Do not add any additional Notes or Explanations.

        User Query: {query}
        """

    r_dict = generate_full_completion(functions_prompt)
    r_tools = json_repair.loads(r_dict["response"])["tools"]
    code = r_tools["tool_input"]["code"]
    response = execute_python_code(code)
    return response

Finally, when you feed it a query like "what's the 12312 * 321?" the whole system whirls into action, the model figures out which tool and code snippet to use, executes it, and bam! You've got your answer.

Just to Show Off

Let’s see it in action with a couple of examples:

# 3952152
print(get_answer("what's 12312 *321?"))
# 500
print(get_answer("how many even numbers are there between 1 and 1000?"))
# Paris
print(get_answer("what's the capital of France?"))

We're blending advanced model integration with practical code execution. Whether you're automating tasks, building out a project, or just playing around to see the capabilities, this setup might just be your next go-to.

And, there you go—a delightful mix of Python, APIs, and some AI magic to streamline how you handle and execute code snippets. As always, tweak, tinker, and tailor it to your needs. Happy coding, everyone!

This was inspired by:

• Trying to open-source my package AgentRun and struggling for a few days with Python tooling

• Jeff Knupp excellent article on how to open source a Python project hitting the 10+ year mark and needing an update

• Simon Willison updated cookiecutter tool python-lib that did most of the heavy lifting with the pain of automating the publishing process to PyPI.

Tools and Concepts

When you're gearing up to open source a Python project, there are a handful of tools and concepts that really come in handy. I'm going to walk you through some of the essentials that I've found invaluable. Keep in mind, my recommendations are based on personal experience, so they might be a bit subjective.

Let's break it down:

• Project Layout: How to structure your files.

• The pyproject.toml File: This is crucial for project settings.

• Pytest: For all your testing needs.

• GitHub Actions: Automate workflows directly from your GitHub repository.

• MkDocs: For awesome documentation.

• PyPI Trusted Publishing: Get your package out there easily.

• Cookiecutter: A lifesaver for starting projects quickly.

• Recipe: Step-by-step guide to get you rolling.

If you're already familiar with these tools, feel free to jump straight to the Recipe section where you can follow the practical steps to get your project live.

Project Layout

When you're open sourcing a Python project, the way you organize your project layout is crucial. It's often the first thing potential contributors or users notice. A cluttered or confusing structure can be overwhelming for newcomers, so it's essential to get it right from the start.

Every project should include at least three key directories:

1. A docs directory for all your project documentation.

2. A directory named after your project where the actual Python package lives.

3. A tests directory to hold all your test files.

On top of these, you'll typically have several important top-level files like LICENSE, README.md, and possibly a few others. However, it's wise to keep the number of top-level files to a minimum. To give you a clearer picture, here's a simplified snapshot of the layout for one of my projects, AgentRun.

├── .github
│   └── workflows
│       ├── publish.yml
│       └── test.yml
├── .gitignore
├── LICENSE
├── README.md
├── agentrun
│   └── __init__.py
├── agentrun-api
├── docs
├── mkdocs.yml
├── pyproject.toml
└── tests
    └── test_agentrun.py

The pyproject.toml File

The pyproject.toml file is a configuration file for Python projects, standardized by PEP 518. It specifies build system requirements and can be used to configure tools like black, isort, and pytest and many others. This file is crucial for modern Python packaging and dependency management.

In the context of open-sourcing a Python package, pyproject.toml plays several roles:

1. Dependency Specification: It declares build dependencies required to compile your package from source. This ensures that anyone trying to build your package from the source will have the right tools installed automatically.

2. Package Metadata: It can include metadata about your package such as name, version, authors, and more. This information is essential for package distribution and maintenance.

3. Tool Configuration: It allows you to configure various tools used during development in a single, standardized file. For example, settings for formatters, linters, and test frameworks can be specified here, ensuring consistency across environments.

4. Build System Declaration: It declares which build backend (like setuptools, flit, or poetry) is used to build your package. This is crucial for reproducibility and compatibility in different environments.

The difficult thing about this part is that you have many options and configurations you can choose from. In the beginning I recommend to keep it simple and gradually add more tools as you are get more comfortable.

Here is a minimal pyproject.toml similar to the one generated by python-lib that I recommend to start with.

[project]
name = "Your project name"
version = "0.1"
description = "Your project description"
readme = "README.md"
requires-python = ">=3.8"
authors = [{name = "your name"}]
license = {text = "the license choosen from the project"}
classifiers = [
    "License :: OSI Approved :: {{ the license name }} "
]
dependencies = []

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project.urls]
Homepage = "your github repo page"

[project.optional-dependencies]
test = ["pytest", "mkdocs"]

This pyproject.toml file doesn't have any dependencies and only pytest and mkdocs for the test version of the package. So, this is just a starting point and would need to be adjusted depending on your own project needs.

Pytest

When publishing an open-source package, it's good to include some tests. This not only invites contributors to your project but also reassures users about the reliability of your package. Without tests, there's a risk that contributors might unintentionally break existing features, and potential users might hesitate to depend on your software.

Pytest is one of the most popular Python testing framework and I recommend starting from the beginning with it.

Let's imagine you have a package that simply increments a number by 1. Here is the code in your package/__init__.py :

def inc(x):
    return x + 1

Then under your tests/ directory. You should have a file test_inc.py that uses Pytest to test your code.

from package import inc

def test_inc():
    assert inc(3) == 4

To run your tests, simply type pytest in the terminal. Pytest will execute all your tests and report back whether they passed or failed. This immediate feedback loop is invaluable for maintaining a robust codebase.

Github Actions

When you're planning to share and collaborate on a project, having a solid CI/CD setup is important. Continuous integration (CI) refers to the practice of automatically integrating and testing code changes into a shared source code repository without breaking anything. Continuous Delivery, automates the release of validated code to a repository following the tests that happen in CI.

At a minimum, you want your code tested automatically every time someone pushes a new change. There are many vendors and tools that can do that, but I like Github Actions. Github is by far the most popular platform to store and share code and having native CI/CD in the same place as your code makes things easier.

You store all your Github actions in a directory called .github/workflows . Here is a simple github action that tests your code every time someone's open a Pull Request or pushes code.

name: Test

on: [push, pull_request]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10"]
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}
        cache: pip
        cache-dependency-path: pyproject.toml
    - name: Install dependencies
      run: |
        pip install '.[test]'
    - name: Run tests
      run: |
        pytest

Mkdocs

MkDocs is a fast, and simple static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file - mkdocs.yml.

There are many plugins, themes, and automation recipes that helps with documenting your package. But to start, you really need only 2 things.

1. mkdocs.yml configuration file

2. index.md file under the docs/ directory

The mkdocs.yml only needs a site name and a site url to be valid. Here is a minimal example:

site_name: My Documentation
site_url: https://example.com

PyPI Trusted Publishing

Your code should now be ready for you to build and distribute to PyPI. The process is quite simple with PyPI Trusted publishing mechanism. You need to sign into PyPI and create a new “pending publisher”. Here is what that looks like.

Trusted publishing is essentially you allowing your GitHub repository your-name/your-package to publish packages with the name your-package via a github action.

The next step is to create a publish.yml in your .github/workflows that uses Github Action to publish your code to PyPI automatically whenever you create a release.

name: Publish Python Package
on:
  release:
    types: [created]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10"]
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}
        cache: pip
        cache-dependency-path: pyproject.toml
    - name: Install dependencies
      run: |
        pip install '.[test]'
    - name: Run tests
      run: |
        pytest
  deploy:
    runs-on: ubuntu-latest
    needs: [test]
    environment: release
    permissions:
      id-token: write
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: "3.12"
        cache: pip
        cache-dependency-path: pyproject.toml
    - name: Install dependencies
      run: |
        pip install setuptools wheel build
    - name: Build
      run: |
        python -m build
    - name: Publish
      uses: pypa/gh-action-pypi-publish@release/v1

Let's simplify this file and see what's happening:

• Essentially, we have two main tasks here: testing and deploying.

• In the deployment section, we're tackling a couple of key activities. First off, we install the necessary tools for packaging—these include setuptools, wheel, and build. Next, we use Python’s build module to actually build the package. Finally, we publish the package to PyPI using the pypa/gh-action-pypi-publish@release/v1 action.

So, in a nutshell, we're testing the code, building it, and then pushing it up to PyPI.

Cookiecutter

Cookiecutter is a command line tool that automates the process of starting a project. Instead of going through all these steps manually, you can just run cookiecutter and the tool will generate all the necessary files and directory structure.

For tools discussed, I tried to follow python-lib structure and recommendations (only adding mkdocs as an extra dependency). So, to achieve the same results you can:

1. Install cookiecutter - pipx install cookiecutter

2. Run it - cookiecutter gh:simonw/python-lib

3. Optional: Add mkdocs as discussed above.

There are many cookiecutter templates around with all kind of different options. The truth is, you really want a minimal well-maintained one.

Cookiecutter is great as a starting point, but relying completely on it without understanding all the tools inside can be painful when things go wrong. That's why I recommend python-lib which gives you the minimum tools you need to publish your package and nothing else.

Recipe

1. Start by creating a PyPI account if you don't already have one. Check to ensure the package name you want isn’t already taken.

2. Next, set up a new GitHub repository with your package name.

3. Install Cookiecutter if you haven't already

   pipx install cookiecutter

4. Run Cookiecutter and answer the prompts to generate your project skeleton

   cookiecutter gh:simonw/python-lib

5. Create a virtual environment in your project directory and activate it.

    cd my-new-library
    # Create and activate a virtual environment:
    python3 -mvenv venv
    source venv/bin/activate
    # Install dependencies so you can edit the project:
    pip install -e '.[test]'
    # With zsh you have to run this again for some reason:
    source venv/bin/activate
    # test the example function
    pytest
   

6. Initialize a Git repository, commit the initial project structure, and push it to GitHub

    git init
    git add -A
    git commit -m "Initial structure from template"
    git remote add origin https://github.com/{{github_name}}/{{repo_name}}.git
    git push -u origin main
   

7. Check the GitHub Actions tab in your repository to confirm that the test workflow is running. This setup ensures your code is tested with every push.

8. On your local machine, create a 'develop' branch for ongoing development.

    git checkout -b "develop"
   

9. Add MkDocs to your development dependencies in the pyproject.toml file.

    # nothing else changed
    [project.optional-dependencies]
    test = ["pytest", "mkdocs"]
   

10. Create a docs directory and add index.md in there. You can copy the info in the README.md to index.md for now.

11. Create a mkdocs.yml file. Here is a minimal example.

    site_name: <your_package> documentation
    site_url: https://<your_username>.github.io/<repo_name>-docs/
    

12. Install mkdocs and deploy a documentation site.

    # update the dependencies
    pip install -e '.[test]'
    # test and adjust the documentations as needed
    mkdocs serve
    # deploy your documentations to github pages
    mkdocs gh-deploy
    # Add site/ to your .gitignore file
    echo "site/" >> .gitignore
    

13. When you are happy with the changes, commit your documentation changes to the 'develop' branch and push

    git add -A
    git commit -m "documentation"
    git push -u origin develop
    

14. For new features, create a feature branch, add your code, and then merge it back into 'develop' when ready

    git checkout -b "new-feature"
    # work on your code until done
    git add -A
    git commit -m "feature complete"
    git checkout develop
    # now we merge the completed feature
    git merge "new-feature"
    

15. When ready to release, merge 'develop' into 'main', pushing the changes to GitHub

    💡

    Don't forget to upgrade your version in pyproject.toml every time you merge develop into main after the initial release.

    git checkout main
    git merge develop
    git push
    

16. On Github - create a new release for your package:

    On your repository’s main page, click on "Releases" which is typically found on the right side of the sub-navigation menu under the code tab.

    Click the “Draft a new release” button or the “Create a new release” link if you don't have any releases yet and fill the information there.
17. On release creation, the publish Github action would run and automatically publish your releases to PyPI

And that's it. Congratulations, you have now published your first open-source Python package with all what you need for future contributors and users.

For a practical example, check out the AgentRun repository, which uses the same tools and processes outlined here.

A few days later, I discovered that GPT-4 access is restricted in several countries, and private usage is quite limited. This prompted me to revisit some old code I had, which was designed to handle email webhooks for incoming messages. I decided to adapt this code to create a solution for these issues because it would be cool.

Thus, Assistant-OverMail was born—an open-source project aimed at simplifying access to GPT-4 via email.

Assistant-OverMail

Assistant-OverMail is an open-source web application designed to simplify your workflow by emailing assistant@overmail.ai (or any domain name if you self-host) and getting a response back from GPT-4.

You can use this to use GPT-4 privately, go around employers' blocks, or simply as a short-cut to deal with emails in the inbox. You can try it at: overmail.ai

For those interested in self-hosting, the setup requires Docker, Docker Compose, and a standard Nginx + SSL configuration. Assistant-OverMail is compatible with any email service provider that supports SMTP, such as Mailgun, SendGrid, or AWS SES. Just provide your SMTP credentials in the .env file to get started.

How does it work?

The main business logic is really around emails webhooks. All mail providers as far as I know provide a way to send you webhooks on receiving emails that match certain rules. Here is an example from my mailgun dashboard.

In our case, whenever an email is sent to "assistant@overmail.ai", mailgun forwards this email as a form POST to a designated endpoint. This endpoint performs several checks before passing the email content to GPT-4 for processing. After generating a response, the system utilizes Django's built-in email capabilities to send this response back to the sender.

Parsing Email Addresses

One new thing I learned from trying to improve my old code is Python's parseaddr. My old code had a big block of code handling all kind of different emails formatting on the FROM field with if statements.

The parseaddr function from Python's email.utils module is used to parse a string into its constituent name and email address parts. It takes a single string argument, which typically represents an email address with an optional display name, and returns a tuple of the form (realname, email_address). Here's a quick breakdown:

1. Input: A string in the format "Display Name <email@example.com>" or just "email@example.com".

2. Output: A tuple where the first element is the display name (if provided, otherwise an empty string), and the second element is the email address.

The function is quite handy for extracting email addresses and names from headers or similar formats in emails where such data may be presented in a more human-readable form.

Payments

Stripe checkouts are relatively straightforward. You generate a URL on the server from your Stripe key and the Price id, and redirect your user to that URL. They pay on Stripe infrastructure, and you get notified via a webhook on success.

I did want to experiment with Bitcoin payments though as something new to learn. This feature, ended up taking almost a week - mostly because I didn't know what I was doing (but now I do). It really deserves its own post, but at a glance - there are three ways you can accept Bitcoin payments over the internet, ranked by effort:

1. Via BTCPay server on your own infrastructure

2. Custodial services APIs (Alby or Strike for example).

3. Via a payment processor on their infrastructure

Since this is a small project that I wasn't looking forward to maintain a server for or build a custom checkout experience on top of Alby or Strike APIs- I ended up going with #3.

OpenNode is a Stripe-like service that essentially offers the same type of workflow for Bitcoin. A checkout URL is generated on the server, and the user is redirected there. They pay on OpenNode infrastructure, and you receive a webhook on success.

Frontend

This is a quite simple project, and so I decided to go with a Carrd template for it. All what was needed is really a form to submit an email address too, and everything else is either payment on someone's else infrastructure or a webhook.

Alternatives

There are a few Zaps from Zapier that offers Email to GPT-4 integration. There is also at least 1 extension that works in the browser to autofill email responses.

I consider this project complete and overall a success. You can find the completed project here: https://overmail.ai and the code - MIT licensed here: https://github.com/Jonathan-Adly/assistant-overmail/