Overview

The Comet platform has extensive LLMOps functionality powered by a specialized SDK, this SDK is refered to as the LLM SDK and is open-sourced at comet-llm.

For more information on LLM support in Comet, see Track LLMs.

Note

The LLM SDK is under active development. If there are any features you would like to see implemented, reach out on Github

Install LLM SDK¶

To use the LLM SDK, downoad and install using:

pip install comet-llm

The full reference documentation is available at LLM SDK reference

Use the LLM SDK to log prompts and responses¶

The LLM SDK supports logging prompts with it's associated response as well as any associated metadata like token usage. This can be achieved through the function log_prompt:

import comet_llm

comet_llm.log_prompt(
    prompt="Answer the question and if the question can't be answered, say \"I don't know\"\n\n---\n\nQuestion: What is your name?\nAnswer:",
    prompt_template="Answer the question and if the question can't be answered, say \"I don't know\"\n\n---\n\nQuestion: {{question}}?\nAnswer:",
    prompt_template_variables={"question": "What is your name?"},
    metadata= {
        "usage.prompt_tokens": 7,
        "usage.completion_tokens": 5,
        "usage.total_tokens": 12,
    },
    output=" My name is Alex.",
    duration=16.598,
)

Use the LLM SDK to log chains¶

The LLM SDK supports logging a chain of executions that may include more than one LLM call, context retrieval, or data pre- or post-processing.

First start a chain with its inputs:

import comet_llm

comet_llm.start_chain({"user_question": user_question})

For each step in the chain, you can create a Span object. The Span object keeps track of the input, outputs, and duration of the step. You can have as many Spans as needed, and they can be nested within each other. Here is very simple example:

with comet_llm.Span(
    category="YOUR-SPAN-CATEGORY", # You can use any string here
    inputs=INPUTS, # You can pass any object in that dict as long as it can be dumped in JSON
) as span:
    YOUR_CODE_HERE

    span.set_outputs(outputs=OUTPUTS) # You can pass any object in that dict as long as it can be dumped in JSON

Here is a more realistic example of Spans including nesting of them:

def retrieve_context(user_question):
    # Retrieve the context
    with comet_llm.Span(
        category="context-retrieval",
        name="Retrieve Context",
        inputs={"user_question": user_question},
    ) as context_span:
        context = get_context(user_question)

        context_span.set_outputs(outputs={"context": context})

    return context


def llm_call(user_question, context):
    prompt_template = """You are a helpful chatbot. You have access to the following context:
    {context}
    Analyze the following user question and decide if you can answer it, if the question can't be answered, say \"I don't know\":
    {user_question}
    """

    prompt = prompt_template.format(user_question=user_question, context=context)

    with comet_llm.Span(
        category="llm-call",
        inputs={"prompt_template": prompt_template, "prompt": prompt},
    ) as llm_span:
        # Call your LLM model here
        result = "Yes we are currently open"
        usage = {"prompt_tokens": 52, "completion_tokens": 12, "total_tokens": 64}

        llm_span.set_outputs(outputs={"result": result}, metadata={"usage": usage})

    return result


with comet_llm.Span(
    category="llm-reasoning",
    inputs={
        "user_question": user_question,
    },
) as span:
    context = retrieve_context(user_question)

    result = llm_call(user_question, context)

    span.set_outputs(outputs={"result": result})

Then finally end your chain and upload it with:

comet_llm.end_chain(outputs={"result": result})

Upcoming features¶

The LLM SDK is under active development, we are currently planning on implementing:

Support for logging prompt and response embeddings

Feel free to reach out on Github with any and all feature requests !

Feb. 9, 2024