In this post we are going to walk through the example of using the scrapegraph SDK with LanceDB and Burr. While Scrapegraph and Burr already have an integration layer, this is a very powerful, simple approach using the scrapegraph SDK.

This is written in collaboration with Scrapegraph, and adapted from the example on their repository.

First we will go over the tools — Burr, Scrapegraph, and LanceDB. We will next walk through the code, show how it looks in production, and provide some further reading.

The tools

ScrapeGraph

ScrapeGraphAI is a Python library for web scraping that leverages LLMs and direct graph logic to build efficient scraping pipelines for websites and local documents, including XML, HTML, JSON, and Markdown files. With over 16k stars on GitHub and an active Discord community, it has become a popular tool for developers. The library also offers a hosted API, enabling access to its most advanced scraping services, including:

• SmartScraper: extracts structured content from any webpage given a URL, a prompt, and optionally, an output schema.

• LocalScraper: works similarly to SmartScraper but is tailored for offline HTML files.

The API is production-ready and includes features such as automatic proxy rotation, anti-bot solutions, and fixed output response verification. In the example below, we’ll demonstrate how to use the Markdownify service from the hosted API. While the service includes a free tier, you can always opt for the OS version.

Burr

Burr is a lightweight Python library you use to build applications as state machines. You construct your application out of a series of actions (these can be either decorated functions or objects), which declare inputs from state, as well as inputs from the user. These specify custom logic (delegating to any framework), as well as instructions on how to update state. State is immutable, which allows you to inspect it at any given point. Burr handles orchestration, monitoring, persistence, etc…).

You run your Burr actions as part of an application – this allows you to string them together with a series of (optionally) conditional transitions from action to action.

Burr comes with a user-interface that enables monitoring/telemetry, as well as hooks to persist state/execute arbitrary code during execution.

You can visualize this as a flow chart, i.e. graph / state machine:

And monitor it using the local telemetry debugger:

While the above example is a simple illustration, Burr is commonly used for AI assistants (like in this example), RAG applications, and human-in-the-loop AI interfaces. See the repository examples for a (more exhaustive) set of use-cases.

LanceDB

(From the LanceDB github) LanceDB is an open-source database for vector-search built with persistent storage, which greatly simplifies retrieval, filtering and management of embeddings.

The key features of LanceDB include:

• Production-scale vector search with no servers to manage.

• Store, query and filter vectors, metadata and multi-modal data (text, images, videos, point clouds, and more).

• Support for vector similarity search, full-text search and SQL.

• Native Python and Javascript/Typescript support.

• Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure.

• GPU support in building vector indices.

• Ecosystem integrations with LangChain, LlamaIndex, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.

LanceDB's core is written in Rust and is built using Lance, an open-source columnar format designed for performant ML workloads. We won’t be leveraging the full extent of its capabilities in this post (scale/multi-modality) — but it makes an ideal candidate for tutorial as it is open-source and easy to get started with.

Chat with your Webpage

Now that we’ve gone over our tools, let’s talk about how we’ll use them! Our goal is to define a flow (Burr application) that:

1. Fetches markdown from webpages (scrapegraph)

2. Chunks the content and stores it in a vector store (lancedb)

3. Allows to query the db and generate an answer using a LLM

We can see all of this as Nodes connected together in a Graph, where the Nodes are the actions we want to perform.

First, we will instantiate a scrapegraph client/set the logging level:

We will then use Burr’s functional API to define some actions. Let’s start by scraping a webpage:

Then, let’s define some helper functions/data models to hold chunks from our page in the lancedb vector store, as well as some tokenizing utility functions to parse our text:

Now, let’s define our action — it creates a local vector store if it can’t find one, and adds the chunks to the chunks table. Note that this is parameterized — the user provides the URL for a webpage, which gets used for scraping:

Our last action will be asking a question of the page — taking in a user query and outputting the response. This is simple — it passes the context into the OpenAI API, and returns the contents of the response.

To ensure visibility, we turn on opentelemetry logging (so Burr can pick up traces), and wire together our actions in an application!

Finally, we can run it, passing the query and the webpage URL:

As you can see, it is completely correct:

To debug, we can examine the trace in the Burr UI — you can see every step of the process, as well as attributes/traces for the lance/OpenAI queries:

Wrapping Up

While we went over a fairly simple example, there’s a lot more we can do. In particular, we can:

1. Bypass loading the page if we already have loaded it

2. Provide more interactive query tooling — ask multiple questions of the same page

3. Log to a Burr cloud instance to get persistence telemetry

4. Additionally, log to any opentelemetry provider

5. Use SmartScraper to extract a summary and some keywords for each chunks to use as metadata in the vector store, to improve retrieval

6. Semantic crawling to fetch multiple webpages from the same domain and chat with the whole website

If you’re interested in reading more, or want to get started on these, don’t hesitate to reach out! Here are some more resources to get you started:

• 🚀 Get your API Key: ScrapeGraphAI Dashboard

• 🐙 GitHub: ScrapeGraphAI GitHub

• ⏩ Burr: Github

• 🛢️ LanceDB: Github

Introduction

Many AI applications start off simple – you query some hosted model, do a little bit of prompt engineering, process/store/index the result, and stream it back to the user. You model this as a simple ping-pong application – (AI → Human → AI, etc…). Inevitably, as you hit some success, you will need more from your application. The natural next step is to enable it to call tools, or dive into a more complicated research question (E.G. with chain of thought prompting). This is when it turns into an agent – an AI system that dynamically directs their own processes and tool-usage, either with or without human input. An agent is often modeled by a graph with state – it has a set of capabilities and needs to dynamically decide the order.

As your agent’s capabilities grow in complexity, you likely start to realize that it needs to execute multiple tasks at once – generate several candidates for user/LLM selection, leverage various tools simultaneously, etc… While you will still model this as a graph, you end up being in multiple states at once, and have to worry about distributing the computation and coordinating state between tasks. Furthermore, you have to now deal with a host of additional challenges — what happens if one or more of your agents fails, or disconnects from the parent processes? How do you provision specific resources and orchestrate execution in a distributed system?

In this post we show how the Burr library helps you solve these problems. In particular, we will be executing parallel sub-agents/workflows on Ray and persisting the state to enable easy restart from failure. We will:

1. Dig into Burr/why we built it

2. Briefly walk through one of Burr’s Parallelism APIs and show an example of generating poems

3. Show how to improve the implementation of parallelism in two ways

   1. Improve reliability through checkpointing

   2. Distributed agent calls on Ray

4. Wrap up – talk through other applications/directions for future work.

Burr

Burr is a lightweight Python library you use to build applications as state machines. You construct your application out of a series of actions (these can be either decorated functions or objects), which declare inputs from state, as well as inputs from the user. These specify custom logic (delegating to any framework), as well as instructions on how to update state. State is immutable, which allows you to inspect it at any given point. Burr handles orchestration, monitoring, persistence, etc…).

You run your Burr actions as part of an application – this allows you to string them together with a series of (optionally) conditional transitions from action to action.

Burr comes with a user-interface that enables monitoring/telemetry, as well as hooks to persist state/execute arbitrary code during execution.

You can visualize this as a flow chart, i.e. graph / state machine:

And monitor it using the local telemetry debugger:

While the above example is a simple illustration, Burr is commonly used for AI assistants (like in this example), RAG applications, and human-in-the-loop AI interfaces. See the repository examples for a (more exhaustive) set of use-cases.

A multilayered API for parallelism

To allow for agent-level parallelism, burr uses recursion as a core primitive. No matter how small the sub-application is, each parallel branch gets run in its entirety under the context of a parent action, as a set of separate, simultaneous burr applications. Conceptually, parallelism in Burr uses a simple map reduce pattern.

While there are many ways to specify the sub-actions (see the docs on parallelism), we will be showing an example of the MapStates API. The user implements a class that takes the state in as an input, splits it out into multiple separate states (one for each task), runs each of those, and joins the results.

To demonstrate, let’s write an agent that takes a prompt and generates a poem in multiple styles from that prompt. To generate the poem, it will delegate to a write/edit loop that does a few iterations to improve the poem.

Our overall structure will look like this – note that this graph does not include the details of the sub-applications:

The user provides workflow-level parameters (poem_types, poem_subject, and max_drafts). From these, we manipulate the state, and generate a sub-application from each step. The most interesting part is the distribution — E.G. the use of MapStates. The code looks like this:

To tie this together, we’ll include this action as part of our overall application:

Now we’ve got all we need to run! We execute as we would any normal application – by default, the sub-applications will be spawned in threads, which will be joined when they are done. This is easy to follow in the UI – you can see an application with multiple child applications.

You can also click to expand them individually:

And view state for the sub-applications inline (in case you wanted an in-depth analysis of a limerick based on a state machine…).

When you click on one of the child applications, you will see a link back to the spawning parent:

You can also annotate the workflow, and track additional data with opentelemetry, giving you full visibility into the way your sub-applications function.

Parallelism improvements

While Burr has capabilities to make launching, waiting for, and joining tasks easy, it additionally provides two advanced APIs to make parallelizing more powerful.

1. Persistence of sub-agents/workflows for fault-tolerance with checkpointing

2. Customization of execution to run on a task orchestrator (such as Ray)

Fault-tolerance with inherited persisters

If the sub-applications fail, we will need to start where it left off – either for a retry or a manual adjustment (of state/code). To enable this, Burr leverages the persister framework. Specifically, it allows child agents/workflows to inherit their parent’s persisters. This is enabled by default in the MapActions, MapStates, and MapActionsAndStates interfaces. It works as follows:

1. The parent task instantiates a set of sub-applications, with a stable ID (this is currently dependent on the task order, but will be customizable in the future)

2. Each sub-application clones the parent application, creating a new application ID with a stable hash of the parent ID and the unique child ID

3. These clones also copy the persister object that the parent has – ensuring that the configuration matches

4. When sub-applications are executed, they persist state according to the persister they instantiated

Thus when an application fails – either when the parent task exits early, when a child fails randomly, or when the connection is lost, it can pick up from where it left off on the next instantiation with the same state. All tasks that are complete will effectively be no-ops (relaunched from a final state), and those that are in progress or have not started will resume at the latest state they persisted.

To ensure that this works, you need to add persistence to your parent application with the initialize_from API – the settings will cascade to the child apps, so that if a parent knows to start from where it left off, the child will as well.

Practically, this looks like this (using the SQLite persister for simplicity — if you’re doing true concurrent systems you’ll likely want a more multi-tennant database):

Note that you can further customize the persistence of child apps by overriding the state_initializer or state_persister implementation in your subclasses.

If you run a regularly failing application in the UI you will see it start from where it left off and eventually complete! In this case we’ve just stuck something flaky in a while loop…

You can also use the Burr UI to dig into the failure reasons with the full stack trace:

Custom Execution with Ray

Subtasks in Burr are, by default, managed with a simple concurrent threadpool (using concurrent.futures.ThreadPoolExecutor). That said, you can actually customize execution with any implementation of concurrent.futures.Executor. To run in Ray, Burr provides an implementation that submits Ray tasks on the user’s behalf, as part of the ray plugin.

To enable, all you need to do is call with_parallel_executor on the application builder, passing it a factory function that instantiates the executor (in this case, the constructor will suffice):

When you run your application, all sub-applications will be tasks on ray! They will write back to the same persister configuration (copied on a per-task basis), so ensure that if you’re doing more than the local testing we’re doing here, you use a truly distributed persister (redis, postgres, mongodb, etc…). Note you will also want to use the s3 based tracking client, or leverage a distributed filesystem, as your local filesystem will likely be unavailable on remote hosts.

Wrapping Up

In the prior post, we outlined Burr’s approach to building parallel agents – demonstrating a few of the APIs to make specifying tasks and joining the results easy. In this post we dug into two more advanced features – ensuring your agent is fault tolerant with checkpointing, and distributing out execution with Ray. While we did not demonstrate further extensions, Burr enables you to build your own concurrent.futures.Executor to run on whatever distributed system you want (dask, runhouse, etc…), or build your own custom persister to match the schema in your DB (please contribute these back if you think they might be general purpose!)

Next steps

While the API is stable and production-ready, we have a host of improvements we are planning. In particular, we want to:

1. Add customizable IDs for sub-applications, both so you can specify equivalence between runs and observe in the UI

2. Add additional asynchronous-based parallelism capabilities – while we handle parallelism with asyncio.gather, we will extend customizable execution capabilities to async as well (E.G. allow async execution on Ray).

3. Consider providing a higher-level execution API (than concurrent.futures) to specify more burr-specific concerns (maximum workers, serialization concerns, messages passed between actions using event-driven/actor model, etc…)

4. Add the ability to execute any task (not just parallel sub-actions) on a remote host using the same executor

And we’re building quickly! If you have a need for these (or other ideas), please reach out. We’re happy to chat.

Further Reading

If you’re interested in learning more about Burr:

• Join our Discord for help or if you have questions!

• Subscribe to our youtube to watch some in-depth tutorials/overviews

• Star our Github repository if you like what you see

• Check out the recent OpenTelemetry integration for additional observability

To build reliable AI software, i.e. wrangle the non-determinism inherent with LLMs, one must graduate from shipping on “vibes” to instead invest in building systematic evaluations. Much like unit test suites ensure standard software is reliable, LLM-powered software also need a suite of tests to directionally assess or evaluate reliability. The hard part is that these tests aren’t as simple as those for regular software.

In this post we show one possible approach to building an LLM application test suite and using it in a test driven manner - with pytest! If you aren’t using python, don’t use pytest, or don’t use Burr, this post will still be helpful at a high level — you can replicate the learnings with other testing libraries/systems. Overall, this post should leave you with:

1. A mental picture of how to iterate on LLM / agent applications.

2. What pytest is and how to use it.

3. What type of extension we need to make pytest work in the LLM / Agent use case.

4. Features of Burr that can help with pytest and creating more reliable AI software.

For the code that’s featured here you can also find it under this pytest example in the Burr repository.

Side note: if this topic interests you Hugo Bowne-Anderson and Stefan Krawczyk have a Maven Course: “Building LLM Applications for Data Scientists and Software Engineers” starting January 6th covering content like this to help you ship reliable AI applications.

Share

This post is a long one, here’s an outline of this post’s structure to help you navigate it (or use the left hand side navigation bar of substack!), e.g. skip the introduction to pytest if you already know it:

1. Why pytest & Burr.

2. The high level SDLC with Burr & pytest.

3. pytest constructs & what needs to be adjusted to use pytest.

4. Burr’s functionality to help create pytest tests & a sketch of the TDD loop.

5. Some tips and then links to a full example.

6. Summary & FAQ.

Why pytest?

Pytest is a very popular python library for functionally testing python software. Rather than introduce a new tool, why not use what’s already available, and something you probably already know how to use? We think that reusing a well known tool will enable a more streamlined iteration loop (read on for more details).

Note: While pytest is very extensible due to its plugins framework, we’ll only be scratching the surface in this post. There are a few ways you could achieve the desired end result. If you come up with something better, please share it!

Subscribe now

Why Burr?

When building LLM Applications / Agents, you get to choose how you build. Do you do everything yourself or do you use a framework? Burr allows you to take both paths. Rather than hide LLM calls (e.g. LangChain), or only require prompts to build agents (e.g. CrewAI), Burr aims to be really good at allowing you to “glue” whatever logic together you need in a standard manner, in a way where iteration is fast and observability is included. This means that you can choose to control all LLM API calls yourself, use other frameworks like Haystack, or do both, when building LLM applications / Agents with Burr. Burr’s strengths are that it solves the pain of (a) how to structure your code for fast iteration, (b) instrumenting for observability.

If you can draw a flowchart of your LLM application / Agent, then your code structure in Burr should mirror it.

What is Burr?

Burr is a lightweight Python library you use to build applications modeled similarly to state machines. You construct your application out of a series of actions (these can be either decorated functions or objects), which declare inputs from state, as well as inputs from the user. These specify custom logic (delegating to any framework), as well as instructions on how to update state. State is immutable, which allows you to inspect it at any given point. Burr handles orchestration, monitoring, persistence, etc…).

You run your Burr actions as part of an application – this allows you to string them together with a series of (optionally) conditional transitions from action to action.

Burr comes with a user-interface that enables monitoring/telemetry, as well as hooks to persist state/execute arbitrary code during execution.

You can visualize this as a flow chart, i.e. graph / state machine:

And monitor it using the local telemetry debugger:

While the above example is a simple illustration, Burr is commonly used for AI assistants (like in this example), RAG applications, and human-in-the-loop AI interfaces. See the repository examples for a (more exhaustive) set of use-cases.

A SDLC with Burr and pytest

With the rise of LLMs, the regular software development cycle (SDLC) doesn’t really work. Why? Well you can’t write a simple test, you need to curate (and continually update) a suite of tests or a data set. This means you need to continually gather data to know how well your application is working and curate existing tests when requirements / the world changes. To put what we’ll present in perspective, here’s an example software development lifecycle (SDLC) with Burr and pytest:

While we don't cover everything in the diagram, in this post we specifically show how to do most of the Test Driven Development (TDD) loop:

1. Create a test case.

2. Run the test case.

3. Create a dataset.

4. Show how you might construct evaluation logic to evaluate the output of your agent / augmented LLM / application.

In terms of the rest of this post, we’ll introduce pytest for those that are new to it, and then explain how we can adopt it for our TDD cycle to wrangle evaluating non-deterministic outputs, i.e. the outputs of an LLM / agent.

pytest Introduction

pytest basics

We like pytest because we think it's simpler than the unittest module python comes with. To use it you need to install it first:

pip install pytest

Then to define a test it's just a function that starts with test_:

Yep - no classes to deal with. Just a function that starts with test_. Then to run it:

pytest test_my_agent.py

Boom, you're testing!

Parameterizing Tests

We can also parameterize tests to run the same test with different inputs. This comes in handy as we build up data points to evaluate our agent / llm calls. Each input is then an individual test that can error. Here's an example:

pytest fixtures

Another useful construct to know are pytest fixtures. A “fixture” is a function that is used to provide a fixed baseline upon which tests can reliably and repeatedly execute. They set up preconditions for a test, such as creating test data, initializing objects, or establishing database connections, etc… To create one in pytest, you declare a function and annotate it:

To use it, one just needs to “declare” it as a function parameter for a test.

We will both be constructing our own pytest fixture in this post, and using some pytest fixtures that come prebuilt with the pytest-harvest library.

Using pytest to evaluate your agent / augmented LLM / application

An agent / augmented LLM is a combination of LLM calls and logic. But how do we know if it's working? Well we can test & evaluate it.

From a high level we want to test & evaluate the "micro" i.e. the LLM calls & individual bits of logic, through to the "macro" i.e. the agent/application as a whole.

But, the challenge with LLM calls is that you might want to "assert" on various aspects of the output / behavior without stopping execution on the first assertion failure, which is standard test framework behavior. So what are you to do?

By default pytest fails on the first assert failure

In the examples we've shown above, the test will stop executing on the first assertion failure. This is the default behavior of pytest. While this works well for regular deterministic software testing, failing at the first failed evaluation for an LLM / agent test would be prohibitive in getting a full picture of our LLM / Agent application’s health. If you’re scratching your head here as to why this would be a problem, this next section is for you.

What kind of "asserts" do we want?

We might want to evaluate, or assert on the output in a number of ways:

1. Exact match - the output is exactly as expected.

2. Fuzzy match - the output is close to what we expect, e.g. does it contain the right words, is it "close" to the answer, etc.

3. Human grade - the output is graded by a human as to how close it is to the expected output. E.g. is this funny?

4. LLM grade - the output is graded by an LLM as to how close it is to the expected output.

5. Static measures - the output has some static measures that we want to evaluate, e.g. length, etc.

It is rare that you solely rely on (1) with LLMs, and you'll likely want to evaluate the output in a number of ways before making a pass / fail decision. E.g. that the output is close to the expected output, that it contains the right words, etc., and then make a pass / fail decision based on all these evaluations.

We will not dive deep into what evaluation logic you should use. If you want more information there, we suggest you start with posts like this. The TL;DR: is:

1. You need to understand your data and outcomes to choose the right evaluation logic.

2. Focus on binary measures, e.g. yes / no, pass /fail, etc.

3. Categorize errors to know what to test / check for

So now what? How do we evaluate all the assertions before making a pass / fail decision?

Not failing on first assert failure / logging test results

As mentioned above, one limitation of pytest is that it fails on the first assertion failure. This is not ideal if you want to evaluate multiple aspects of the output before making a pass / fail decision.

One approach is using the pytest-harvest plugin to log what our tests are doing. This allows us to capture the results of evaluations in a structured way without breaking at the first asserting failure.

This doesn’t limit how we use pytest. Rather, it enables us to mix and match appropriate hard "assertions" - i.e. definitely fail, with softer ones where we want to evaluate all aspects before making an overall pass / fail decision. We will walk through how to do this below using a few pytest & pytest-harvest constructs.

pytest-harvest

pip install pytest-harvest

pytest-harvest is a plugin that provides a number of pytest fixtures. These fixtures have certain behaviors that we’ll use to log and then see the results of evaluations from within our tests.

results_bag is a fixture that we can log values to from our tests. This is useful if we don't want to fail on the first assert statement, and instead capture a lot more. E.g. you can assign arbitrary values to “keys” like below.

We can then access what was assigned in the results_bag from the pytest-harvest plugin via the module_results_df fixture that provides a pandas dataframe of the results:

We can then evaluate them as we see fit for our use case. E.g. we only pass tests if all the outputs are as expected, or we pass if 90% of the outputs are as expected, etc. You could also log this to a file, or a database, etc. for further inspection and record keeping, or combining it with open source frameworks like mlflow and using their evaluate functionality.

Note: we can also combine results_bag with pytest.mark.parametrize to run the same test with different inputs and expected outputs:

Using Burr's pytest Hook

With Burr you can curate test cases from real application runs. You can then use these test cases in your pytest suite. Burr then has a hook that enables you to curate a file with the input state and expected output state for an entire run, or a single Burr action. See the Burr test case creation documentation for more details on how (or see the “reproduce” tab in the Burr UI like below).

Once you run the command you’ll get some code and a JSON file that you can then manage and curate into a pytest test. Then to combine with the pytest-harvest fixtures to capture evaluations, we need to add the `results_bag` fixture argument to the python function — note the final code below:

To test an entire agent, we can use the same approach, but instead rely on the input and output state being the entire state of the agent at the start and end of the run.

Using the Burr UI to observe test runs

You can also use the Burr UI to observe the test runs. This can be useful to see the results of the tests in a more visual way. To do this, you'd instantiate the Burr Tracker and then run the tests as normal. A note on ergonomics:

1. It's useful to use the test_name as the partition_key to easily find test runs in the Burr UI. You can also make the app_id match some test run ID, e.g. date-time, etc.

2. You can turn on opentelemetry tracing to see the traces in the Burr UI as well.

3. In general this means that you should have a parameterizeable application builder function that can take in a tracker and partition key.

TDD: Adopting an iterative SDLC to get out of POC purgatory

Unlike traditional software where writing tests means you’re done, with LLMs that is just the start of an iterative loop. By logging traces to get data and turning them into datapoints for tests in pytest, one can move fluidly between an application development, test driven development, and back. As more datapoints are accumulated, you can then better evaluate and measure prompt changes and their impacts.

By using pytest as the means to systematically go from shipping on vibes to having a structured way to evaluate and ensure that projects are heading in the right direction, you stand a much better chance of getting your LLM & Agent applications out of POC purgatory.

Your TDD Loop with Burr and pytest

To put it more concretely, the above code showed how you could:

1. Pull an application trace and create a pytest test for it.

2. Add and manage datapoints with parameterize and/or Burr’s pytest integration.

3. Evaluate multiple aspects of a single test and then aggregate them for multiple data points.

With this setup, you can now operate a Test Driven Development (TDD) loop to optimize and work on a single LLM call! As you make changes to the code / prompt, you’ll be able to use pytest to exercise that one LLM call and then evaluate the changes in a systematic fashion, without working only off of vibes.

Some uses of TDD:

Here are some more situations where applying TDD can help speed up development:

1. Guardrails. Taking the aforementioned TDD approach can make sure that prompt injection, harmful content, etc. is ignored, or dealt with appropriately.

2. Evaluating the impact of using a different LLM. If you got your application working with a large model, and want to assess if you can switch to a lower cost model, having a pytest suite you can run will help you quickly assess where you need to tune / change things.

One trick we like - run things multiple times

LLMs are inherently non-deterministic. So one way to explore and determine how the variance of a single prompt + data input leads to different outputs, is to run it multiple times.

With pytest this can take the form of a test that runs an action multiple times, then aggregates the responses to see how different they are. Using this approach can help you better tweak prompts to reduce variance.

The output of this test will show you where variance occurs, and is also then a useful measure in of itself in how stable things are, e.g. maybe it’s okay for some variance to exist, but not too much…

Capturing versions of your prompts to go with the datasets you generate via pytest

As you start to iterate and generate datasets (that's what happens if you log the output of the dataframe), you will need to tie together the version of the code that generated the dataset to the dataset itself. This is useful for debugging, and for ensuring that you can reproduce results (e.g. which version of the code / prompt created this output?). One way to do this is to capture the version of the code that generated the dataset in the dataset itself. Assuming you’re using `git` for version control, one just needs to capture the git commit hash of the code that generated the data set, i.e. the prompts + business logic. If you treat prompts as code, then your workflow might be:

1. Use git to commit changes.

2. Create a pytest fixture that captures the git commit hash of the current state of the git repo.

3. When you log the results of your tests, log the git commit hash as well as a column.

4. When you load / look at the data set, you can see the git commit hash that generated the data set to tie it back to the code that generated it.

To use it we add the fixture to the function that saves the results:

A Runnable Example

If you want to see some working code, using a dummy medical diagnosis workflow. Please see this example. In it you’ll find:

• some_actions.py - a file that defines an augmented LLM application (it's not a full agent) with some actions. See image below - note the hypotheses action runs multiple LLM calls in parallel.

• test_some_actions.py - a file that defines some tests for the actions in some_actions.py.

• conftest.py - a file that contains some custom fixtures and pytest configuration for running our tests.

You'll see that we use the results_bag fixture to log the results of our tests,and then we can access these results via the module_results_df fixture that provides a pandas dataframe of the results. This dataframe is then saved as a CSV for uploading to google sheets, etc. for further analysis. You will also see uses of pytest.mark.parametrize and Burr's pytest feature for parameterizing tests from a JSON file.

To run the tests, you can run them with pytest:

pytest test_some_actions.py

After running the tests, you can see the results in a CSV file called results.csv in the same directory as the tests. Users then generally export / import that to a UI of their choice (e.g. google sheets, or a notebook) for further error analysis and human review.

You should also see the following in the Burr UI that can aid in reviewing some traces.

Share

Summary

In this post we showed how to build a test driven development (TDD) loop using Burr and pytest. Specifically, we showed how you can take a trace from a Burr application and create a pytest case, then use pytest-fixtures (from pytest-harvest) to record all test evaluations and later post-process them to provide an aggregate measure to help you discern the reliability of your AI application over time. This allows you to quickly set up a test driven development loop where you then tweak prompts / code and then run pytest to systematically run evaluations on your changes.

If you like this post, or find other / better ways to use pytest, please leave a comment or create an issue in Burr’s repository. There’s also a FAQ list below.

Leave a comment

Maven Course: Building LLM Applications for Data Scientists and Software Engineers

If you got this far and want to learn more, then sign-up for Hugo Bowne-Anderson’s and Stefan Krawczyk’s Maven Course: “Building LLM Applications for Data Scientists and Software Engineers”. It stars January 6th and will cover topics such as this that are required to ship reliable AI software.

Subscribe now

Further Reading

If you’re interested in learning more about Burr:

• Find the code for this example here

• Join our Discord for help or if you have questions!

• Subscribe to our youtube to watch some in-depth tutorials/overviews

• Star our Github repository if you like what you see

Frequently Asked Questions:

General Questions

Q: Do I need to use Python to implement this testing approach?

A: While the examples use Python, pytest, and Burr, the general principles of systematic evaluation and test-driven development can be implemented in any programming language with appropriate testing frameworks. The key concepts of capturing test results, evaluating multiple aspects of LLM outputs, and maintaining test suites are language-agnostic.

Q: How is this different from traditional software testing?

A: Traditional software testing typically focuses on deterministic outputs with clear pass/fail criteria. Testing AI applications requires handling non-deterministic outputs, evaluating multiple aspects of responses, and often using fuzzy matching or LLM-based evaluation. This approach allows for capturing and analyzing various quality metrics rather than just binary pass/fail results.

Testing Implementation

Q: How do I decide which aspects of LLM output to evaluate?

A: Consider evaluating:

• Content accuracy (exact or fuzzy matching)

• Output format compliance

• Response consistency across multiple runs

• Domain-specific requirements

• Safety and appropriateness checks

The specific aspects will depend on your application's requirements.

Q: What's the recommended approach for handling test data?

A: Start by capturing real application runs using Burr's tracking functionality. Curate these into test cases, focusing on diverse scenarios and edge cases. Maintain separate test suites for different aspects (e.g., content generation, safety checks, format compliance) and regularly update them as requirements evolve.

Performance and Scale

Q: How does this testing approach impact development speed?

A: While setting up systematic testing requires initial investment, it typically saves time in the long run by:

• Catching issues early in development

• Providing clear metrics for improvement

• Enabling confident iteration on prompts and model interactions

• Reducing manual testing effort

Q: Can this testing approach handle large-scale AI applications?

A: Yes, the approach scales well because:

• Tests can be parameterized to handle many test cases

• Results can be exported for analysis in external tools

• The Burr UI provides visibility into test runs

Best Practices

Q: How often should these tests be run?

A: Consider running:

• Core functionality tests with each code change

• Comprehensive evaluation suites daily or weekly

• Stability tests (multiple runs) during significant prompt or model changes

• Performance benchmarks on a regular schedule

Q: How should I maintain test cases over time?

A: Best practices include:

• Regular review and updates of test cases

• Tying together version control with the results of tests.

• Documentation of evaluation criteria

• Tracking of model and prompt versions

• Regular analysis of test results to identify patterns

Troubleshooting

Q: What should I do if test results are inconsistent?

A: Consider these steps:

1. Use the multiple-run approach described in the post to measure output variance

2. Review and possibly adjust evaluation criteria

3. Check if the inconsistency is expected LLM behavior or a bug

4. Consider adjusting prompts to reduce variance if needed

Q: How can I debug failed tests effectively?

A: Utilize:

• Burr's UI for visualizing test runs

• The detailed results captured in results_bag

• pytest's debugging features

• The CSV export functionality for detailed analysis

Integration and Tools

Q: Can this testing approach integrate with CI/CD pipelines?

A: Yes, you can integrate these tests into CI/CD pipelines. Consider:

• Running core tests on every PR

• Scheduling comprehensive evaluations

• Setting appropriate thresholds for test success

• Exporting results for tracking over time

Q: What tools are recommended for analyzing test results?

A: Common tools include:

• Spreadsheet applications for basic analysis

• Jupyter notebooks for detailed investigation

• Visualization tools for tracking metrics over time

• MLflow for experiment tracking

• Custom dashboards using exported CSV data

Burr-Specific Questions

Q: What's the learning curve for implementing Burr in an existing AI application?

A: Burr is designed to be lightweight and integrates with existing Python applications. The main concepts to understand are state machines, actions, and transitions. If you're already familiar with pytest, the additional Burr-specific testing features should be straightforward to adopt.

Q: Can Burr handle asynchronous operations in AI applications?

A: Yes, Burr supports asynchronous operations and parallel execution of actions, as demonstrated in the medical diagnosis workflow example where multiple hypotheses are generated in parallel.

Many AI applications start off simple – you query some hosted model, do a little bit of prompt engineering, process/store/index the result, and stream it back to the user. You model this as a simple ping-pong application – (AI → Human → AI, etc…). Inevitably, as you hit some success, you will need more from your application. The natural next step is to enable it to call tools, or dive into a more complicated research question (E.G. with chain of thought prompting). This is when it turns into an agent – an AI system that dynamically directs their own processes and tool-usage, either with or without human input. An agent is often modeled by a graph with state – it has a set of capabilities and needs to dynamically decide the order.

As your agent’s capabilities grow in complexity, you likely start to realize that it needs to execute multiple tasks at once – generate several candidates for user/LLM selection, leverage various tools simultaneously, etc… While you will still model this as a graph, you end up being in multiple states at once, and have to worry about distributing the computation and coordinating state between tasks.

In this post we show how the Burr library helps you solve these problems. We will:

1. Dig into Burr/why we built it

2. Walk through the API layers burr exposes for parallelism

3. Show a simple example of generating multiple poems in different styles and presenting them to the user

4. Wrap up – talk through other applications/directions for future work

Burr

Burr is a lightweight Python library you use to build applications as state machines. You construct your application out of a series of actions (these can be either decorated functions or objects), which declare inputs from state, as well as inputs from the user. These specify custom logic (delegating to any framework), as well as instructions on how to update state. State is immutable, which allows you to inspect it at any given point. Burr handles orchestration, monitoring, persistence, etc…).

You run your Burr actions as part of an application – this allows you to string them together with a series of (optionally) conditional transitions from action to action.

Burr comes with a user-interface that enables monitoring/telemetry, as well as hooks to persist state/execute arbitrary code during execution.

You can visualize this as a flow chart, i.e. graph / state machine:

And monitor it using the local telemetry debugger:

While the above example is a simple illustration, Burr is commonly used for AI assistants (like in this example), RAG applications, and human-in-the-loop AI interfaces. See the repository examples for a (more exhaustive) set of use-cases.

A multilayered API for parallelism

To allow for agent-level parallelism, burr uses recursion as a core primitive. No matter how small the sub-application is, each parallel branch gets run in its entirety under the context of a parent action, as a set of separate, simultaneous burr applications. Conceptually, parallelism in Burr uses a simple map reduce pattern.

The parent action broadcasts the state to multiple sub-applications (run/controlled by the parent action), each of which has their own state – a transformation of the parent’s state. These are run, waited for, and the result is merged to form a final state for the parent action, which downstream actions consume. While the spawning application and the sub-applications it spawns are largely decoupled, Burr framework provides two features to make this process easier:

1. An API allows for cascading of telemetry from the parent to the child application – the UI is aware of the parent-child relationships, and provides bidirectional links + inline expansions in the trace view

2. Burr provides tooling to make the process of launching tasks, joining/merging state, and tracking linkage automatic.

For (2), Burr has three different API layers – the implementation of each leverages those below. They are:

1. Full control

With the lowest-level API layer, you launch and manage your own tasks, linking them together with the tracker. You are responsible for forking/joining processes, wiring persistence/tracking, etc… to the child applications, and setting up sub-applications. While we won’t go over this in the blog post, you can read about it in the docs.

2. Task-based map/reduce

You can also specify a series of tasks. You did this by implementing the TaskBasedParallelAction class, which produces a generator of SubGraphTasks, each of which specifies the state/application ID for an input. While you still have to wire through some application-level concerns (tracking/persistence), it handles execution. You then specify how to reduce/join the state with a simple reduce function. Again, we will not cover this in the post. Read more in the docs.

3. State/action-based map/reduce

You can specify either a set of actions or a set of states to map over. If you specify both, it will take the cartesian product of the actions/states. There are three different classes to implement – MapActions, MapStates, and MapActionsAndStates (cartesian product). We will be showing an example of the MapStates implementation during the rest of this – hold tight!

Which to choose?

It all depends on how much control you need, but you should choose the greatest layer of abstraction that effectively models your problem. Many agentic shapes we’ve found can be represented as a simple map over either actions/states, which is why we made it the top-level interface. Otherwise, if you want a full set of executions for an experiment/comparison, you’ll likely leverage the cartesian product, and if you have specific logic that doesn’t boil down to actions/states, you’ll use the task-based approach. The fully-controlled API is reserved for odd cases in which the map/reduce pattern does not effectively fit. E.G. if you’re streaming parallel results out, have a few longer-running tasks and want to leverage task inter-communication, etc… For the latter two interfaces, you can customize execution approach, tracking behavior, persistence behavior, and more.

Generating Poems in Parallel

Let’s go over a fairly simple example of the MapStates API – we will be writing an agent that takes a prompt and generates a poem in multiple styles from that prompt. To generate the poem, it will delegate to a write/edit loop that does a few iterations to improve the poem.

Our overall structure will look like this – note that this graph does not include the details of the sub-application:

The user provides workflow-level parameters (poem_types, poem_subject, and max_drafts). From these, we manipulate the state, and generate a sub-application from each step. The most interesting part is the distribution — E.G. the use of MapStates. The code looks like this:

To tie this together, we’ll include this action as part of our overall application:

Now we’ve got all we need to run! We execute as we would any normal application – by default, the sub-applications will be spawned in threads, which will be joined when they are done. This is easy to follow in the UI – you can see an application with multiple child applications.

You can also click to expand them individually:

And view state for the sub-applications inline (in case you wanted an in-depth analysis of a limerick based on a state machine…).

When you click on one of the child applications, you will see a link back to the spawning parent:

You can also annotate the workflow, and track additional data with opentelemetry, giving you full visibility into the way your sub-applications function.

Wrapping Up

Comparison

Burr’s approach towards parallelism differs from other libraries, largely due to the design goals. While langgraph took a heavier-handed pregel-based approach to orchestrate running actions over state, Burr aims towards simplification. The core philosophy is central management and execution – launching tasks, delegating to a single node, and joining the results. While talking to our users, we found that the vast majority prioritized clear control flow primitives that enabled them to reason about what their agents are doing and where. By not taking a state-convergence based pregel approach and then breaking its assumptions to allow a human-in-the-loop control flow, we think our implementation is simpler to understand top to bottom. You could already roll your own parallelism in a Burr action manually, so with this new functionality we focused on simplifying that and integrating parallelism deeply within the Burr UI for a better observability and user experience.

Next steps

Although we covered the basics of parallelism, there is still quite a bit more to cover. In particular – what happens when you lose track of your execution halfway through (E.G. accidentally hit ctrl-c during execution)? Can you ensure all tasks will restart where they left off? Furthermore, multithreading won’t always work – especially if you have anything that requires specialized compute. In the next post, we will talk about how to make this truly distributed with Ray, and ensure that it is fault-tolerant – upon failure, it can resume where it left off.

Further Reading

If you’re interested in learning more about Burr:

• Find the code for this example here (note this is the same as the next post)

• Join our Discord for help or if you have questions!

• Subscribe to our youtube to watch some in-depth tutorials/overviews

• Star our Github repository if you like what you see

• Check out the recent OpenTelemetry integration for additional observability

Introduction

Data-driven decision-making is a useful addition to most businesses. However, a common obstacle to adoption is the slow development-to-production lifecycle; while the initial building stage often involves more effort, the ongoing adjustment of dataflows to new demands should not require a full rewrite. Hamilton creates expressive dataflows to keep data pipeline lifecycles short by improving code readability and maintenance ease.

An important challenge in designing Hamilton was to address the trade-off between asset visibility and function cluttering, which makes code unmaintainable and unreadable. We represent the data pipeline as nodes in a Directed Acyclic Graph (DAG) and designed an internal decorator lifecycle framework to perform DAG-level manipulation. Nodes provide the necessary interface layer abstracting away from Python functions and, in conjunction with the decorator lifecycle, reducing code verbosity.

In this post, we talk about a few things:

1. How Hamilton, can improve your dataflow building capabilities and increase team productivity.

2. How Hamilton naturally represents the trade-off between explicitness and verbosity.

3. How Hamilton uses decorators to overcome that trade-off and make everyone more productive.

4. How Hamilton implements the fine-grained details of the architecture and decorator lifecycle.

If you just want to get a sense of why you want to use Hamilton, read (1) and (2). If you want to understand some of its more advanced features, read (3). If you want to dig into the architecture of those and really want to nerd out (like I do), read (4).

The Power of Explicitness

Working with data is messy; you never know what comes out or which direction you are meant to go. This entails a lot of experimentation, visualization, and hypothesis testing before you arrive at useful business insights. If you need to do this once a year, it may be acceptable to rewrite a previous analysis. Imagine now, that you need to redo the process every day, or better yet, every hour. God forbid, your colleague asks you to share your Jupyter notebook with them.

It would be good to keep track of how things change so that you can retrace your steps and, more importantly, your colleagues know what is happening / can work on the same data pipeline or re-use parts for their needs. This is where some battle-tested principles from software engineering come in handy, such as version control and breaking things apart into single functionality modules.

We can achieve this by making data transformations explicit and modular. This allows us:

• to group them conceptually together, for example creating a feature platform

• finding a good balance between building fast vs. to last, for example, having enterprise grade data pipelines

• incorporating flexibility for future requirements, for example by decoupling the different pipeline stages

The aim is to turn a messy Jupyter notebook used for data analysis into a structured data pipeline (organized in Python modules). This naturally leads to better code readability and to the codebase being better maintained. Hamilton is an opinionated but lightweight framework that enables exactly that.

What is Hamilton?

Hamilton is a standardized way of building dataflows, a.k.a. “pipelines”, in Python. The core concepts are simple – you write each data transformation step as a single Python function with the following rules:

1. The name of the function corresponds to the output variable it computes.

2. The parameter names (and types) correspond to inputs. These can be either passed-in parameters or names of other upstream functions.

This approach allows you to represent assets in ways that correspond closely to code, naturally self-documenting, and portable across infrastructure.

After writing your functions, you call a driver to execute them – in most cases this is a simple import/run (specifying the assets you want computed and letting the framework do the rest), but it provides options to customize execution:

Visit tryhamilton.dev for a quick interactive introduction in the browser.

Explicitness/Verbosity Trade-Off

Hamilton's main (initial) selling point is explicitness. For every asset you have a corresponding function -- modifying that asset is an O(1) operation (a simple lookup), which can help iteration/debugging speed. On the other hand, this represents an increase in the verbosity of your code. To demonstrate:

A common use case would be to pull some dates from a database and calculate the date ranges. For simplicity, let us only consider three dates leading to three date ranges that we need to input into a series forecast, and, depending on the region, the date format changes. Here is the code snippet that only uses plain Python functions and Hamilton to build our DAG.

Keeping things explicit lets us have an overview of the different assets used in that data flow. However, the increase in verbosity can actually harm code readability and maintenance.

A simple solution to decrease verbosity would be to move the database calls to “helper functions” (using an underscore prefix in the Python function tells Hamilton to ignore it in the DAG creation process). We can then create a node that takes as input the region, fetches data from the database, and converts it into datetime objects. With this, we are already able to decrease the verbosity a bit.

This improves the readability a lot! As a first pass at building your dataflow, this can be “good enough” and just using these basic functionalities can serve you for a long time.

Two points to note here; firstly, we reduced code verbosity at the expense of reducing explicitness since we hid the database calls. Secondly, we still have a lot of “empty functions” for the three dates each just returning a single value of a dictionary – imagine we have a dictionary with 20 entries!

It is time to ask: Can we refine the code further by moving core programming concepts such as loops and conditionals to the explicit DAG level?

Overcoming the Trade-Off: Using Decorators to Perform DAG Operations

We are looking for a solution that remains explicit (easy to read/plain English/easy to debug), but it is also much easier to develop/iterate on. It should both cut down on verbosity and allow for abstractions.

Hamilton does this with Python decorators. We are aware that decorators are an advanced concept in Python, but at the same time, we believe we made using them quite simple and provide lots of documentation.

In case you are not familiar with decorators in Python, they are functions that take as an argument an existing function, modify it in some way, and output a new function. See: https://realpython.com/primer-on-python-decorators/ for a detailed explanation. Hamilton uses decorators to attach instructions on how the local subgraph arising from the decorated function should be modified and has its own compiler that uses this data in the DAG creation process (see (4) for a more detailed explanation).

Let’s continue iterating on the same example and refactor the “empty functions” using @extract_fields. This directly corresponds to having a loop on the DAG level (for other concepts such as if/else conditionals and re-using nodes, check out the BONUS section at the end where we use multiple decorators at the same time). We get the following code:

Great, with a single decorator we cut down on verbosity by eliminating three functions while maintaining the DAG structure. Several things happen in the background that we will dive into later, but for now, notice how adding a new date for example involves only adding an entry to the attached decorator that creates a node out of it.

Likewise, the module is less cluttered which makes it easier to understand. It also allows us to quickly add or remove assets based on our needs, which accelerates the development-to-production cycle. Lastly, with fewer functions, we have a smaller code area to cover and our testing process is simplified.

While Hamilton provides you with a powerful, high-level language to build dataflows, you don't have to (and probably shouldn't) jump straight into the deep end. Rather, you can start like above with plain Python and helper functions. Once you see the code getting verbose/messy, you can refactor by creating conditional branches or making the code DRY) by defining common data transformations and decorating the corresponding asset functions - Hamilton will do the rest and construct the correct DAG for you. If you are interested in what happens in the background, we will be digging into Hamilton’s decorator framework for the rest of this post -- buckle in! We're going to go very low-level.

The Nitty-Gritty Details

Before we look at decorators in detail, let us take a step back and understand Hamilton under the hood. In the background, Hamilton abstracts away from user-defined functions and associates them to an object class Nodes, which in turn connect based on names and types, forming a graph. Based on the requested outputs, Hamilton only selects necessary nodes connecting the two endpoints and runs the minimal DAG. This abstraction layer provides the necessary interface we need to use decorators for graph operations.

Let us first go through the high-level overview of how Hamilton creates a DAG (see Figure 1). We start with defining the DAG (user input):

1. Create a Python module and define functions inside.

2. Use Hamilton decorators on some of the functions.

3. Specify the Python module and optional config file for the Hamilton Driver that creates the DAG.

In the last step Hamilton does the following:

1. Collect all functions from the provided module. Skip functions that start with an underscore (i.e., helper functions).

2. Loop through the collected functions and for each function determine which decorators were used.

3. Go through the list of used decorator classes in order of priority and transform the subgraph of that function.

4. Return the subgraph (can be empty).

5. Connect all the subgraphs together into a DAG and validate it.

Great, now that we have the big picture let us explore how Hamilton creates subgraphs and, in particular, how the decorator framework is implemented. At the heart of the decorator lifecycle is the NodeTransformLifecycle abstract class. The class defines the high-level interface/functionality all child classes need as well as implements the call method of the decorator, which watermarks the decorated function with the specific decorator.1

This implementation, where we attach the decorator to the function and delay its execution, allows us to decouple the concrete user instance and subgraph transformations. In other words, the user first gets to define what subgraph transformation he wants to perform before the DAG even exists and delays the execution until we build that part of the DAG and the operations make sense to execute.

If there are no decorators attached to the function, a simple subgraph consisting of a single node corresponding to the underlying function gets created.

Otherwise, for a function with decorators, the decorators get executed in order of subgraph transformation relevance each taking in the output of the prior subgraph transformation:

1. NodeResolver: determine if the subgraph should be present.

2. NodeCreator: construct subgraph node(s) out of the underlying function.

3. NodeInjector: inject a subgraph before the current node.

4. NodeExpander: expand the current node into multiple new nodes.

5. NodeTransformer: transform the output of the current node.

6. NodeDecorator: add metadata to the current node.

This precedence order is critical for chaining together multiple subgraph transformations (stacking decorators) or informing the user that certain operations are incompatible with one another. For example, we can hardly expand the current node into many others if no current node exists.

In the following, we describe each subgraph transformation class and provide a use case. To view all supported subgraph transformations please see the decorators reference.

NodeResolver

The first decorator class resolves whether nodes should be created or not. It is the first decorator class called in the lifecycle since it determines the if/else conditionals for the whole subgraph arising from other decorators on that function.

A prime example is to decorate a function with @config.when, which creates a key-value pair that is used at DAG creation runtime to determine if we short-circuit the lifecycle and output an empty list (no nodes associated with this function).

In this case, we have two different transforms: “transform__a” and “transform__b”. For “transform__a”, the NodeResolver gets called with the provided config and returns the node associated with it. Since Python does not allow two functions to have the same name (and we access all functions in a module by inspecting it) we differentiate the two functions by appending a double underscore namespace that will get stripped when resolved. For the “transform__b”, the NodeResolver returns an empty list (no node is created) since the condition is not fulfilled. We can select which transform (either a or b) gets executed by providing the correct config to the driver at runtime.

In case we do not use the decorator, we would need to define an additional node collecting “transform__a” and “transform__b” with the body performing an if/else statement based on the additional config input.

NodeCreator

The decorator class creates a subgraph consisting of at least one node out of the current function. It is the first decorator class creating a subgraph that all the consequent decorator classes apply transformations on. In case the function has no additional decorators, this class still gets called to create a node out of the function.

For example @dataloader allows us to specify that a certain function is meant to be an I/O interface.

Below we have a node loading JSON data and an additional node created for its metadata (materializer). In case we do not use the decorator, the additional node disappears.

NodeInjector

The decorator class creates a subgraph and prepends it to the existing node via a specific function argument 1-to-1 map. This allows us to pre-process that argument before using it in the function. The preprocessing can be a complicated subDAG, like @with_columns, or a straightforward linear chain of nodes created by @pipe_input.

We take the argument “input_x” as an input argument for “_add_1”, which in turn is input into “_add_2”. We have injected a chain of 2 nodes that modify “input_x” and the result gets injected into “feature” in place of the original “input_x” argument.

In case we do not use the decorator, we could add the transformation as helper functions, but we lose lineage information or connect the nodes the conventional way forsaking the ability to reuse them.

NodeExpander

The decorator class creates a subgraph out of the existing nodes in a 1-to-many fashion. This operation allows us to avoid “empty functions” and can be thought of as a for-loop.

For example, when we have elements of a dictionary, it is cumbersome to manually create a function for each column. Instead, we can use [@extract_fields](https://hamilton.dagworks.io/en/latest/reference/decorators/extract_fields/) to create nodes out of the individual entries.

We created the three nodes “a”, “b”, and “c” and only need to specify their type to have them represented as nodes and avoid the necessity to define individual functions for each of the nodes.

In case we do not use the decorator, we are forced to manually define the three functions and do the extraction in each of them.

NodeTransformer

This decorator class also creates a subgraph and appends it to the existing nodes. It allows us to post-process a node output. For example, @pipe_output takes the output of the function and appends a linear chain of nodes to it, which is the dual behavior of @pipe_input.

We have the same example as with @pipe_input; in contrast, the execution order with @pipe_output is first executing “feature”, which in this case is an identity function, then we apply “_add_1” and afterward “_add_2”.

In case we do not use the decorator, we could similarly use helper functions inside the existing function or connect the nodes the conventional way forsaking the ability to reuse them.

NodeDecorator

The last decorator class in the lifecycle allows us to attach metadata to the subgraph. For example, we can use @tag to add tags to nodes providing code ownership within a team and can filter the nodes that have the specific tag “team=Platform”.

Bonus

Before we wrap up, we just want to have fun and share a more exotic use of decorators. Remember the initial mini-pipeline example of fetching dates and calculating periods between them?

We can STACK decorators to encapsulate all of the above ddmmyy functions as a single function? Practically, we can:

1. Set an if/else conditional for creating a node using @config.when.

2. Create a separate node for each of the dates by using @extract_fields.

3. Convert the string from the database into Python’s datetime format by post-processing all three “a”, “b”, and “c” nodes at once using @pipe_output.

Here is the code that does exactly that:

Selecting the correct region at DAG creation eliminates the unnecessary nodes leaving us with a single region DAG. We have achieved full explicitness with a minimal increase in verbosity!

Adding a new date format involves only adding a Python function for that asset and giving it the appropriate decorators. We managed to further localize the data transformations and shorten the development-to-production cycle. The functions are concise and not cluttered with if/else statements which makes the Python module clean and easy to understand. Lastly, by keeping truly essential functions that describe our dataflow and being able to re-use some of the transformations, we can test for dataflow coherence and do not need to unit test every single node.

Summary

We have shown how Hamilton utilizes decorators to cut down on code verbosity while keeping explicitness intact. Hamilton strikes a balance that lets you maintain a neat codebase that can be shared across users, while at the same time offering the ability to quickly iterate on dataflows. Although Hamilton’s approach requires basic familiarity with Python decorators, it abstracts away the complicated bits and lets you enjoy the full power in a straightforward application.

While the decorator lifecycle framework addresses an important trade-off, we have identified some limitations that warrant a revision of the initial design ideas.

For example, decorators only operate on the current function and keep things very local. While creating new nodes, the decorator does not get any information from other nodes in the DAG, which makes it harder to define a global operation. First attempts at something like that are made with the @mutate decorator.

Another emerging issue is the choice to create new nodes for actions arising from decorators. This can quickly lead to the original DAG being cluttered with additional nodes, which complicates the overview.

If you have not yet used Hamilton, hopefully this post shows that the bar to entry is set very low (you need to use Python in your data pipelines) and once you are comfortable with the basics, have fun exploring the rich and continuously expanding decorator ecosystem. If you are already a seasoned Hamilton user, give us a shout about your experience using decorators and we will incorporate your feedback in our next major overhaul of the decorator lifecycle framework!

1. Package code + dependencies into a container

2. Create API endpoints to interact with the agent

3. Create a client to interact with the endpoints reliably

4. Manage resources and batch requests for efficient processing

5. Monitor API usage

6. Track how your agent behave

7. Persist data produced by the agent for debugging and create training data

8. Add hooks and alerts for certain agent behavior

9. … and more

Burr is an open source framework to build LLM agents and all types of applications that make (or help make) decisions. BentoML is an open source library specialized for deploying LLM, ML, and AI services. Together, they provide the application and serving layers to solve the aforementioned problems.

Find the full example on GitHub

Application layer

With Burr, you create an Application by defining all the possible actions and the transitions between them. The result is graph that’s easy to reason about, making your application easy to reason about and debug.

Actions can read from / write to a centralized state. By storing the state, you create checkpoints you can resume from. This enables “time-travel” debugging where you move backwards through states to see where things went wrong.

When building the Application, you can directly add the persistence, hooks, and telemetry necessary for production via the ApplicationBuilder. Then, you can run your application step, iterate over actions, or run until a halting condition, synchronously or asynchronously.

Burr comes with the open source Burr UI, which allows you to track live execution, inspect past runs, annotate data, create tests fixtures, and more.

Serving layer

BentoML allows you to define a service using a Python class. Its methods, sync or async, will automatically be converted to API endpoints. You can also specify the number of workers and the resources (e.g., CPU, RAM, GPU) required for your service. It also offers autoscaling based on service traffic.

AI, LLM, and ML inference is challenging because preprocessing inputs and making predictions is often an expensive operation. The main way to improve throughput and reduce latency is to batch requests. You could do this manually, but BentoML provides adaptive batching out of the box!

After defining your service class, you want to configure how to package your service. You need to specify the files to include, the Python requirements, and environment variables to read from the host. BentoML uses this configuration to efficiently build a Docker image, and it will automatically update it when required.

Now, run this command to launch your service:

bentoml serve .

BentoML is also capable of automatically generating a client for your service. By creating a SyncHTTPClient object pointed to the service URL, you get a client object with methods that matches the service definition.

With the same deployment and client, you can query the API asynchronously by using a task queue. For instance, instead of calling .run(), you would submit a task via .run.submit() and wait for the task status to be completed and get the result.

Subscribe now

Deployment options

Burr and BentoML are two open-source projects that you’re free to self-host. Alternatively, BentoCloud can host your deployment and provide an interface with permissions, secret management, logs and status, and service versioning. When deploying in the cloud, Burr will need a destination to persist state and send telemetry. Here’s the details for setting up Burr’s UI on AWS S3.

Conclusion

The path to deploying LLM agents is paved with engineering challenges. Good tools should minimize the number of engineering problems to solve and let you focus on the aspect unique to your business that generate value. With Burr for the application layer and BentoML for serving, many challenges are solved before you encounter them yourself.

We want to hear from you!

If you’re excited by any of this, or have strong opinions:

🍱 BentoML Slack (GitHub)

🤖 Burr Discord  (GitHub)

📚 More Burr examples

From proof-of-concept to deployment and maintenance, your technical needs will evolve. To build a reliable product, you will inevitably need custom code and some frameworks may get in the way more than others.

This post will show how a RAG feature typically evolves using three versions of the same project. We will use the 2-layer approach to handle the growing complexity and ensure that we create a maintainable solution without ready-made components.

Follow along on GitHub or Google Colab

The 2-layer approach

The 2-layer approach separates the low-level implementation (layer 1) from the high-level application logic (layer 2). Layer 2 encodes “how the system should make decisions and behave” and Layer 1 processes the data to make these decisions possible.

Burr is a Python framework specifically focused on Layer 2. It structures an Application as a set of actions, transitions & conditions between actions, and state shared between actions. It comes with useful features for deployment such as state persistence, telemetry, annotations, and more. Then, Layer 1 is encapsulated in individual actions (for another example of these concepts in action see our Burr + Haystack post).

Version 1: a simple LLM pipeline

It may be counterintuitive, but you should start with the high-level logic and determine the intended behavior of your application. Skipping these initial questions usually leads to refactoring and slows down development.

Layer 2: Define the Application logic

In the next snippet, we outline the logic to ingest a blog post and ask questions over its content. Note that we haven’t implemented @actions yet, but we get the ability to build and view the graph.

Layer 1: Implement individual @action

The previous outline sets “contracts” for each actions:

1. ingest_blog downloads an HTML page, parses it into a plain text.

2. ask_question prompts an OpenAI LLM with the full blog in the prompt and the user query.

It allows us to build simple implementations for each to get a functional prototype. Notice that @action functions (Layer 1) can use any Python library and don’t hide implementation behind complex objects. This makes it easier to adapt to evolving product requirements.

We can reuse the previous ApplicationBuilder, but we will pass the new _v1 functions instead

Here’s the code to launch your Burr application

Version 2: Let’s use RAG

While Version 1 simply ingested the blog and prompted the LLM, Version 2 will add RAG capabilities. We have to ask ourselves:

Are we making changes to layer 1 or 2?

In this case, we’re focusing on Layer 1. With Burr, we can keep the same “contracts” and simply upgrade the action implementation. As you make changes, you might decide to merge/split actions, modify state variables, etc. Then, you can revisit and update your application outline.

Layer 1: Adding RAG

In the next snippet, you'll notice:

• we define a TextDocument model to create the schema of our LanceDB table. It specifies to embed the text field using the OpenAI model.

• because text chunks are stored on disk, we don't need to pass them via the Application state.

Layer 2: Tracking and observing your application

When developing AI systems, it’s essential to track how your application behave across versions. It’s not uncommon that new exciting techniques don’t fully deliver on promises and it’s important to assess if your earlier and simpler versions perform as well. This is a Layer 2 concern and should be decoupled from your Layer 1 implementation.

Burr makes tracking easy. Simply add a tracker to the ApplicationBuilder. This is compatible with OpenTelemetry! The next snippet will enabling tracking for our RAG application and gather telemetry events from OpenAI and LanceDB.

Now, running your application will emit events that you can see in the Burr UI

Burr UI has many other useful features:

• view token usage

• annotate state values and logged attributes

• create test fixtures from application state

• and more

Subscribe now

Version 3: Keeping your code modular

You probably noticed that functions in Version 2 started to be lengthy and do several things. It’s a good occasion to refactor to smaller functions that will facilitate development, testing, and maintenance.

Layer 1: Modular actions

In the next snippets, we refactor actions using Hamilton, a lightweight library to structure data transformations as directed acyclic graphs (DAGs). Hamilton uses the function and parameter names to infer the dependencies between functions and the graph structure.

action: ingest_blog

While this may seem trivial, these simple functions are easier to test, debug, and modify. It also allows us to develop and test actions outside of Burr. Hamilton has a great caching feature that can speed up development significantly when iterating over your application.

action: answer_question

We encourage considering prompt as code, which allows us to commit and version our prompt with the rest of our code. For instance, the function `system_prompt()` helps understand what information goes into the prompt and an additional docstring can add context to it.

Layer 2: Using Burr + Hamilton

Since Burr is not opinionated about the implementation of `@action` functions, you can simply call Hamilton from it. We add the `OpenTelemetryTracer` to the Hamilton `Driver` to get tracing in Burr UI.

You'll see that the Layer 2 becomes much lighter and only the high-level logic remains.

Find more simplifications in the full example notebook

The biggest benefit of Burr + Hamilton is the unbeatable observability you get. The Burr UI will show a granular breakdown of the operations.

Hamilton even has its dedicated Hamilton UI that tracks execution, catalogs data transformations, and provides in-depth introspection. We're looking to integrate the two further together!

Conclusion

We presented the 2-layer approach to building a RAG application, which separates high-level logic from the implementation of individual actions.

The key lesson is that you should adopt frameworks incrementally. A tool shouldn't lock you in and limit the evolution of your application. Adopting Burr from the start helps you develop in a principled way, and adding Hamilton as the complexity increases helps with maintainability and observability.  

We want to hear from you!

If you’re excited by any of this, or have strong opinions

📣 Join our Discord 

⭐️ us on GitHub

Instructor lets us get structured outputs from LLMs reliably, and Burr helps create an LLM application that's easy to understand and debug. It comes with Burr UI, a free, open-source, and local-first tool for observability, annotations, and more!

This post is a cross-post from a guest post on the Instructor blog and expands on an earlier one: Analyzing Youtube Transcripts with Instructor.

🤖 The full code example is available on GitHub

Generate flashcards using LLMs with Instructor

pip install openai instructor pydantic youtube_transcript_api "burr[start]"

1. Define the LLM response model

With instructor, you define Pydantic models that will serve as template for the LLM to fill.

Here, we define the QuestionAnswer model which will store the question, the answer, and some metadata. Attributes without a default value will be generated by the LLM.

This example shows several instructor features:

• ue to prevent the LLM from hallucinating the value

  • id generates a unique id (uuid)

• The type annotation SkipJsonSchema also prevents the LLM from generating the value.

  • youtube_url is set programmatically in the application. We don't want the LLM to hallucinate it.

• Field can set constraints on what the LLM generates.

  • min_items=3, max_items=5 to limit the number of potential answers between 3 and 5

  • ge=0, lt=5 to limit the difficulty between 0 and 5 with 5 being the most difficult

2. Retrieve the YouTube transcript

We use youtube-transcript-api to get the full transcript of a video.

Generate question-answer pairs

Now, to produce question-answer pairs:

1. Create an instructor client by wrapping the OpenAI client

2. Use .create_iterable() on the instructor_client to generate multiple outputs from the input

3. Specify response_model=QuestionAnswer to ensure outputs are QuestionAnswer objects

4. Use the messages to pass the task instructos via the system message, and the input transcript via user message.

This will return an generator that you can iterate over to access individual QuestionAnswer objects.

Create a flashcard application with Burr

Burr uses actions and transitions to define complex applications while preserving the simplicity of a flowchart for understanding and debugging.

1. Define actions

Actions are what your application can do. The @action decorator specifies what values can be read from or written to State. The decorated function takes a State as first argument and return an updated State object.

Next, we define three actions:

• Process the user input to get the YouTube URL

• Get the YouTube transcript associated with the URL

• Generate question-answer pairs for the transcript

Note that this is only a light refactor from the previous code snippets.

2. Build the Application

To create a Burr Application, we use the ApplicationBuilder object.

Minimally, it needs to:

• Use .with_actions() to define all possible actions. Simply pass the functions decorated with @action.

• Use .with_transitions() to define possible transitions between actions. This is done via tuples (from_action, to_action).

• Use .with_entrypoint() to specify which action to run first.

3. Launch the application

Using Application.run() will make the application execute actions until a halt condition. In this case, we halt before process_user_input to get the YouTube URL from the user.

The method .run() returns a tuple (action_name, result, state). In this case, we only use the state to inspect the generated question-answer pairs.

You can create a simple local experience by using .run() in a while loop

Next steps

Now that you know how to use Instructor for reliable LLM outputs and Burr to structure your application, many avenues open up depending on your goals!

1. Build complex agents

Instructor improves the LLM's reasoning by providing structure. Nesting models and adding constraints allow to get facts with citations or extract a knowledge graph in a few lines of code. Also, retries enable the LLM to self-correct.

Burr sets the boundaries between users, LLMs, and the rest of your system. You can add Condition on transitions to create complex workflows that remain easy to reason about.

2. Add Burr to your product

Your Burr Application is a lightweight Python object. You can run it within a notebook, via script, a web app (Streamlit, Gradio, etc.), or as a web service (e.g., FastAPI).

The ApplicationBuilder provides many features to productionize your app:

• Persistence: save and restore State (e.g., store conversation history)

• Observability: log and monitor application telemetry (e.g., LLM calls, number of tokens used, errors and retries)

• Streaming and async: create snappy user interfaces by streaming LLM responses and running actions asynchronously.

For example, you can log telemetry into Burr UI in a few lines of code. First, instrument the OpenAI library. Then, add .with_tracker() the ApplicationBuilder with a project name and enabling use_otel_tracing=True.

3. Annotate application logs

Burr UI has a built-in annotation tool that allows you to label, rate, or comment on logged data (e.g., user input, LLM response, content retrieved for RAG). This can be useful to create test cases and evaluation datasets.

Conclusion

We've shown how Instructor helps getting reliable outputs from LLMs and Burr provides the right tools to build an application. Now it's your turn to start building!

We want to hear from you!

If you’re excited by any of this, or have strong opinions:

📣 Join our Discord 

⭐️ us on GitHub

A few key considerations are:

• How efficiently can I test a new idea or build a demo?

• Can I reliably move my project to production?

• How easy is it to make changes as project requirements evolve?

Haystack and Burr can help you build LLM pipelines and agentic workflows, respectively. In this post, we explore how well they do on these criteria and how you can use them together to build fast and reliably.

Haystack 101: build a pipeline

Haystack is a framework for building data pipelines, especially RAG. You start by creating Components then you assemble them into a Pipeline and run it.

This snippet defines a simple RAG application (adapted from this Haystack tutorial)

Haystack provides a catalog of ready-to-use Component (LLM APIs, vector databases, document processors, data readers, etc.) helps get started quickly. It encourages a “LEGO brick” approach to building pipelines (the example only 40 lines of code) and makes it easy to swap components and try different configurations. Haystack also enables you to define custom components (documentation).

We also maintain Hamilton, a lightweight framework to define pipelines. It is broader than RAG / LLMs, but we have several examples on the topic.
The product WrenAI actually uses both Haystack and Hamilton under the hood!

Burr 101: build an application

Burr is a framework for building applications and agents. It has a similar feel to Haystack, you define Actions (equiv. Component) and transitions between them to form an Application (equiv. Pipeline). We’ll explore their differences next.

The previous RAG pipeline could be recreated entirely in Burr. Burr is flexible as to how you define Actions, you could even use an Haystack Component inside one as shown in embed_text().

Differences between Haystack and Burr

Centralized state

With Haystack, you build a Pipeline by having data flow from a component directly to another. With Burr, your Application has a centralized State, which facilitates writing logic referencing earlier events.

Practically, it means you can always save the State of an application and load it later to resume. If you encounter a bug, can move through previous states to find what went wrong. Then, you can edit State values or modify the Action’s code to fix the issue and resume. Burr also provides utilities to create test cases from failed State to avoid this issue in the future.

Decisions and transitions

To develop an agent capable of complex behaviors, we need to express conditions under which an action should be taken.

Haystack creates pipelines with set entry and exit points. By using, Routers you can implement conditional logic and dispatch data to different components. The ConditionalRouter uses Jinja2 for safety, but limits the expressiveness of routing conditions.

Burr was built specifically for agents, state machines and “decision-making” applications. You can add a Condition on any transition in your Application. The transition can read the State and execute a decision function to determine the next action (documentation).

The next snippet illustrate an agent that searches the web to generate an answer. By adding a when() condition on the transition between verify_citation and search_the_web, we can loop if the generated answer has factual errors (as detected by the verify_citation) action.

Path to production

As a pipeline framework, Haystack can be deployed and orchestrated in various ways. The Hayhooks library allows to serve your Haystack pipeline as an API endpoint. For monitoring, Haystack can generate OpenTelemetry events (note: an Openlemetry destination is required) and offers structured logging.

For Burr, the ApplicationBuilder object allows to instrument your application to meet production needs. Here are some examples:

• .with_tracker() log execution and optionally enable OpenTelemetry events, which can be viewed in the open source and local-first Burr UI.

• .with_hooks() allows to add custom logic on action execution (e.g., send a Slack notification)

• .with_typing() use Pydantic to validate the State and action outputs.

• .with_state_persister() store the State to disk and resume later (e.g., allow a user to resume its session)

• and more

The following snippet creates an application that will log execution with OpenTelemetry under the burr-rag > demo in the Burr UI and send alerts on Slack when exceptions occur.

The Burr UI allows you to monitor your application in real time and view past application logs created by the tracker and OpenTelemetry. It helps with debugging and provide other insights such as the number of tokens used and the estimated cost. Burr UI also ships with a data annotation tool. This tool is free, open source, and comes with Burr, when you install the appropriate target:

pip install "burr[tracking]"

Subscribe now

Haystack + Burr integration

With the new integration, you can merge the catalog of modular components from Haystack and the application-building features of Burr.

While this was possible before, it was verbose. Haystack code needed to be wrapped in a Burr action and the Component inputs/outputs mapped to State. To illustrate, this retrieve_documents() action reads values from State, passes them to Component.run(), and outputs are written back to State:

The new HaystackAction allows you to the same Burr action succinctly. Simply pass the action name, the Haystack Component, and State fields it reads and writes from/to. Your Burr application can mix HaystackAction and regular actions.

Here's the same action as above using the HaystackAction:

Now, you’re able to benefit from Haystack's catalog of components to build Burr applications quickly and reliably. All Burr features become available at almost no additional cost compared to a regular Haystack Pipeline definition. For existing Haystack users, you can fully reuse your custom components for building agents.

What if you already have Haystack pipelines?

You can convert your Haystack Pipeline into a Burr Application in a few lines of code. The function haystack_pipeline_to_burr_graph() converts the pipeline into a Burr Graph that you can use to build the application.

The next snippet converts the pipeline defined at the beginning of this blog:

What if I want to integrate other frameworks?

You can always use your favorite framework (Haystack, Hamilton, LangChain, LlamaIndex, etc.) within a Burr Action. We’re looking forward to add similar integrations to streamline Burr development. Let us know which one to implement next!

As your project evolve, you can swap the Haystack Component used in your HaystackAction, use a custom Haystack Component, or replace the HaystackAction by a regular Burr action with arbitrary Python code. This is a common step as you adjust and upgrade production Burr applications. With Burr you’re not backed into a corner, you’re free to leverage any framework you like as requirements evolve.

Conclusion

On your next project, determine if you’re building a pipeline or an agent and think about your future requirements for production. Haystack and Burr are two complimentary frameworks that can help and their clean integration can provide a lot of value at a low cost. Should you need to customize and extend beyond your initial application, Burr has the constructs and the flexibility so you can iterate with confidence.

We want to hear from you!

If you’re excited by any of this, or have strong opinions:

📣 Join our Discord 

⭐️ us on GitHub

Why is it so hard to get AI systems into production? Among many reasons, you may be haunted by the fear of catastrophic hallucinations and strange behavior. While AI is dressed up to seem human-like, it is prone to errors similar to any other computer program. Specifically, if you’re not exact about what you want, it can produce troubling behavior. Cases of this have resulted in high-profile stories about companies that have proudly released their AI-driven features to production, only to be on the receiving end of lawsuits and unhappy customers. See, for example, the recent Air Canada lawsuit.

The downside is that it may never be possible to solve these problems perfectly – AI, at its current state, is neither precise enough nor appropriately incentivized to reliably *not* mess things up. The upside, however, is that you can be both proactive and reactive to these issues enough to feel confident in releasing your product. While there are multiple approaches to be proactive with Burr, we’re going to be talking about effectively reacting in this post.

The following strategies are table-stakes for anyone who wants to leverage AI seriously, E.G. in any part of their business that produces some value:

1. Full monitoring – know everything that happens, when it happens, and to whom

2. Further investigation – be able to dig into the data and react/save for later if anything goes wrong

3. Intervention – make alterations to the data/reach out to the impacted users

4. Retuning – take the mistakes an app made and ensure it doesn’t happen again

We’re going to be focusing on (2), (3), and (4). Specifically, how to mark data for further review, react to it, and use that data to retune the behavior of your application. For (1), see our original post on Burr and the Burr UI. No need to read these, however, as we’ll first conduct a quick overview of Burr.

Burr

Burr is a lightweight Python library you use to build applications as state machines. You construct your application out of a series of actions (these can be either decorated functions or objects), which declare inputs from state, as well as inputs from the user. These specify custom logic (delegating to any framework), as well as instructions on how to update state. State is immutable, which allows you to inspect it at any given point. Burr handles orchestration, monitoring, persistence, etc…).

You run your Burr actions as part of an application – this allows you to string them together with a series of (optionally) conditional transitions from action to action.

Burr comes with a user-interface that enables monitoring/telemetry, as well as hooks to persist state/execute arbitrary code during execution.

You can visualize this as a flow chart, i.e. graph / state machine:

And monitor it using the local telemetry debugger:

While the above example is a simple illustration, Burr is commonly used for AI assistants (like in this example), RAG applications, and human-in-the-loop AI interfaces. See the repository examples for a (more exhaustive) set of use-cases.

Annotations in Burr

For the sake of this post, we’ll be using the “tool-calling” example from a prior blog post. No need to read it – the high-level is that we used the tool/function APIs to build an agent that chose between response modes and used prebuilt functions to answer as intended. We will be marking data we think is funky, reviewing it, then turning it back into a tuning set. One of the tools we will use will be query_weather, which takes in a latitude/longitude and gives out a free-form description of the weather – we’ll be using this as an example.

Reviewing

The new annotations feature in the UI allows you to mark steps in an application for any reason – E.G. say they’re a hallucination, flag them for review, marking the response as a good decision, etc… To do this, you click on the + symbol in the annotations column in a trace view, which opens up a dialog:

Annotations are identifiable by tags – these are free-form strings (with a set of common default values), that can help identify/query the data. An annotation consists of one or more observations, each on a specific data point (attribute, state field, etc…). These observations have a thumbs up/thumbs down capability, as well as free-form text and an (optional) ground truth. This allows for multiple observations (each associated with a datapoint).

You can then browse the annotations you created for your application with the “annotations” tab on the right hand side

Reacting

In the case of troubling behavior, you want to be able to react quickly. Burr’s annotation feature lets you query data by tags/action name, enabling you to use the tags as a queuing system for handling annotations. Say, for the sake of argument, that someone named Elijah is your trusty QA associate:

Back to the prior example, say we decide on a tag schema with the following tags:

1. hallucination

2. incorrect

3. correct

4. to-review:<name> (E.G. to-review:elijah) or reviewed-by:<name> (E.G. reviewed-by:elijah)

5. urgent

In order to quickly react (and thus provide quality customer service), we can filter all the created annotations by tag at a project level. You can further review, modify tags, and edit annotations.

So, the workflow we anticipate is:

1. The developer/on-call person gets a complaint (or does a routine audit)

2. They immediately mark as to-review:elijah 

3. Elijah then goes through annotations marked as to-review, and reviews accordingly (marking them as reviewed-by:elijah

Retuning

While we’ve talked about how to react to errors in your AI applications, annotating Burr application runs also gives you the capability to proactively change the behavior for later.

You can do this through two sets of provided tools:

Adding unit tests

To do so, you can click on the annotation in the project view, and you’ll get a full application view. You can then click on the “reproduce” tab, which will give you a command to create a test case.

See further documentation here on how to tune this.

Downloading data for fine-tuning

Click on the “Download” button next to the search bar to download the CSV of all relevant data! You can massage that to fine-tune your next model, train a new model from scratch, or have a dataset to send for debugging.

Wrapping up

In this post we showed how to leverage the Burr UI to quickly react to problematic behavior in your AI applications. We showed:

1. How to annotate/tag data

2. How to review that data

3. How to use that annotation data for evaluation/fine-tuning down the line

Leveraging these approaches can give you confidence on your application — both that you can quickly react to any negative user experiences and use that data to ensure they don’t happen in the future.

Next Steps

While the annotations feature is currently very powerful, we plan to implement the following:

1. Centralized schemas for tags, so that teams can more effectively collaborate

2. More advanced search capabilities for annotations

3. User-aware/team-specific queuing (related to (1))

And we’re building quick! Please reach out if you need any of these features or want to provide feedback, we’re building with use-cases in mind.

Further Reading

In the meanwhile:

• Join our Discord for help or if you have questions!

• Subscribe to our youtube to watch some in-depth tutorials/overviews

• Star our Github repository if you like what you see

• Check out the recent OpenTelemetry integration for additional observability

• See the code for this post among our examples

With release 1.79.0 Hamilton now offers caching of execution results! With caching enabled, executing a node will store results on disk along with some metadata. On subsequent executions, it will automatically skip execution of any previously executed steps by loading results. To enable it, simply add .with_cache() to your Builder

from hamilton import driver
import my_pipeline

dr = (
    driver.Builder()
    .with_modules(my_pipeline)
    .with_cache()
    .build()
)

Here are some scenarios where caching is a clear win:

1. Faster development iterations (in scripts or notebooks)

2. Avoid cost of repeated LLM calls and embeddings during agent development

3. Incrementally load data by skipping previously processed files

4. Recover from errors during debugging a data pipeline, starting where you left off

This blog will detail why we built caching and how it works. Then, we’ll illustrate some of these use cases that caching can help you with! First, let’s go over the basics of Hamilton (skip the next section if you’re already familiar).

Subscribe now

Hamilton 101

Hamilton is a standardized way of building data pipelines in Python. The core concepts are simple – you write each data transformation step as a single Python function with the following rules:

1. The name of the function corresponds to the output variable it computes.

2. The parameter names (and types) correspond to inputs. These can be either passed-in parameters or names of other upstream functions.

This approach allows you to represent assets in ways that correspond closely to code, naturally self-documenting, and portable across infrastructure.

Then, you create a Driver from this dataflow definition and you can use it to execute code. In the next snippet, we request the node C for external_input=12.

The pain of expensive operations

If you're working in data (e.g., data science, machine learning, ETL, BI, LLM applications) you know the feeling of launching a script or a notebook cell, and having a minute+ to get results.

Unless there's a bug midway that makes the job fail... Hopefully, it happened before that expensive operation (SQL join, model training, downloading large data, sending a batch of paid LLM requests) 💸💸 Or maybe the job did complete, but there are obvious mistakes in the produced data... Unsure, you prefer running the job twice to check if results match 🤞

load dataset >> chunk >> embed >> evaluate RAG

To alleviate this pain, you can start to create custom checkpoints by saving intermediary results and adding some logic to load checkpoints if available.

load dataset >> chunk >> embed > checkpoint > evaluate RAG

However, as your project grows in complexity, so does your checkpointing code. It’s more to manage. Also, checkpointing logic can be difficult to generalize between dev/prod and across projects in general.

What is caching?

Caching broadly means "reusing stored results instead of repeating operations”, in other words, it is automated checkpointing. To make this possible, Hamilton versions the code that defines the node and the values passed to it.

For example, let’s take this 3 node dataflow and focus on processed_data()

After executing it once, its result will be stored. On the second execution, the node will be skipped (cache hit) if it receives the same values for raw_data or cutoff_date.

It would be a cache miss if:

• the value for raw_data is different

• the value for cutoff_date is different

• the function processed_data() was edited (e.g., change the currency conversion from 0.73 to 0.75)

Caching in Hamilton

To get started with caching in Hamilton, just add .with_cache() to your Driver.

To get more information about execution, you can retrieve the logger and set the level to INFO or DEBUG

There's also a new visualization included via the Driver.cache attribute

dr.cache.view_run()

Hamilton’s caching comes with effective defaults, its behavior can also be tuned node-by-node:

• the “caching behavior”: default, always recompute, default, ignore (and more)

• how to version data for specific Python types

• how to create cache key

• where to store metadata and results (could be different places, by project, globally)

Providing this amount of control to the user allows for optimal execution.

To learn more about caching in Hamilton, feel free to complete the tutorial (GitHub, Colab) or review the documentation.

Caching in notebooks

Caching pairs nicely with the Hamilton notebook extension, which allows you to define your dataflow in a notebook cell (see tutorial).

Say you're developing the above retrieval-augmented generation (RAG) application. It involves multiple steps (loading, parsing, chunking, etc.) with many potential approach for each step. If you try a new embedding step, you'll be able to automatically skip the upstream loading, parsing, and chunking!

Another benefit of caching is that you can safely restart the notebook kernel and shutdown your computer after a long day of work. When you'll come back, you can resume from where you left!

Subscribe now

Caching with dynamic dataflows

Hamilton provides the constructs Parallelizable and Collect to create multiple branches of the same processing steps (learn more). For example, you can retrieve HTML pages for multiple URLs and extract articles from it.

The image below shows a dataflow where url “expands” in many branches that will be collected at collect_chunked_url_text. Each branch will execute: article_text, chunked_text, and url_result

When using Parallelizable/Collect with caching, branches are cached individually. Consequently if there's an error with the 7th URL, the first 6 sucessful branches will skipped on retry. Similarly, if you successfully executed 10 URLs and add 2 more, only the new ones will be executed. This can be a useful pattern for incremental loading.

Caching for debugging

Caching can also be a helpful tool for debugging your Hamilton dataflow. Let’s say you’re running your dataflow via a script and it fails midway through. If you have caching enabled, you can open an interactive session or a notebook, create a new Driver and start debugging.

from hamilton import driver
import my_module

dr = (
    driver.Builder()
    .with_modules(my_module)
    .with_cache(path=...)  # point to the cache used by the script
    .build()
)

You can call dr.execute() to see if the error is reproduced. From there, you can modify the dataflow in my_module and try re-executing, and steps unchanged will be automatically skipped! Hamilton also provides structured logs and entry points to manually inspect metadata and results produced by the cache (learn more).

We’ll soon be supporting caching on AWS, GCP, Azure, and more. This will allow the same debugging pattern of dataflows deployed via Airflow et al. Reach out to us if you’re interested in this feature - find us in Hamilton’s slack.

We want to hear from you!

Caching is a new core feature of Hamilton that we want to improve and expand. It already benefits many use cases, but we want to know your experience and ideas!

If you’re excited or have strong opinions, join our Hamilton community on Slack . It’s also a great place to get help with Hamilton!

📝 leave us an issue if you find something

⭐ give us a star on the Hamilton repository if you like what we’re working on!

We recently launched Burr to create LLM agents and applications.

A typical development workflow for each model includes pulling features from our Snowflake data warehouse, engineering some features manually, and implementing a standard set of pandas dataframe transformations. This leads to a lot of time spent modifying Snowflake queries and repeated code that results in a very isolated feature engineering pipeline that’s not easily accessible to other data scientists. The reusability of our features is low, and the relative universe of features considered for feature selection is small due to the overhead in trying to explore new features. 

We knew we could increase efficiency for our data scientists, minimize train-serve skew, and unlock new use cases that hadn’t been explored yet. Our goal was clear: we needed a centralized platform to support and promote the reuse of ML features across multiple models. This is not a problem unique to our business, and feature stores and commercial feature platforms clearly tout themselves as the solution to these problems. 

We evaluated and completed proof-of-concepts with the top five commercial and open-source providers in the market.  While each platform had its strengths, none were able to fully meet our needs. Ultimately, we chose to build our own in-house solution, leveraging existing infrastructure and integrating open-source tools including Hamilton. 

Our Feature Platform Requirements

As we set out to find the right feature platform, we developed a list of requirements to help us focus on what we really needed and to evaluate each platform consistently. At a high level, the list included capabilities like: 

• Connects to our existing data sources (Snowflake, S3, Postgres, etc.), provides flexibility for any other source we may have, and allows joining across data sources

• Enables calculating on-demand features for real time inference

• Supports pre-computing features to both an online and offline feature store, in batch and streaming contexts 

• Generates accurate training data using point-in-time joins 

• Supports windowed aggregations 

• Handles complex chained transformations 

• Is Python-native and makes it easy to express features as code

Why No Platform Checked All the Boxes

Our requirements are fairly standard and what you’d expect any feature platform to provide. However, the main lesson we learned was that while a platform may check the boxes on paper, it often fell short during our proof-of-concept. Specifically, we had challenges with:

1. Flexibility: Some platforms supported only a limited set of data sources, or restricted us to a single data source at a time. Several platforms allowed only a narrow range of Python packages for transformation logic, with rigid support for custom packages and environments. As our business grows, we didn’t feel we’d be able to adapt quickly enough by waiting on a third party to add necessary functionality.

2. Scalability: Our data tends to be very wide (1000s of columns), and no platform was able to process it more efficiently than our existing infrastructure and pipelines. Our on-demand feature engineering requires very low latency, and there was often additional serving overhead compared to what we achieved with our DIY prototypes. 

3. Complex Transformations: We currently manage tens of thousands of features, and calculating them often involves a complex directed acyclic graph (DAG) of operations to transform raw data into output features. Only one platform we tried really solved this problem, and we realized how key it was to the solution we needed. 

Why We Chose to Build In-House

As a company with a strong foundation in data engineering, we believed we had the expertise to build a better, more tailored solution in-house. The key factors that drove this decision were:

1. Control and Flexibility: We needed to be able to design and evolve the system to meet the specific requirements of our models. By building it ourselves, we can iterate and optimize as needed without waiting on a vendor’s roadmap. 

2. Integration with Existing Infrastructure: We already have a complete set of tools at our disposal. Building in-house allows for seamless integration with our Kubernetes clusters, Snowflake data warehouse, and enterprise event bus, without requiring us to refactor everything to fit into a rigid commercial solution. 

3. No Vendor Lock-in: We were honestly hoping to find a perfect solution we’d feel comfortable locking in to. Building a feature platform is no easy feat, and we recognize that every vendor is solving difficult problems, but for now, retaining flexibility is a key advantage.

Enter Hamilton: The Right Framework for the Job

During our evaluation, we discovered Hamilton, an open-source Python framework designed for creating flexible, modular, and portable data pipelines (think everything from data transforms, to ML, to GenAI). It has a growing community on slack, with weekly releases and a meet-up group, and is driven by the DAGWorks Inc. team that built the internal ML Platform at Stitch Fix. What caught our attention was how Hamilton’s approach to expressing transformations, i.e. features in our context, aligned perfectly with our goals of flexibility and scalability.

1. Modular and Declarative: Hamilton allows us to define extremely complex feature transformations in a modular way. We have several pipelines already which compute 500 – 2,000 features each, including multi-layered dependencies between features.  Each feature is defined as a function, with clear dependencies and no hidden state. This makes our pipelines far more maintainable, testable, and faster to build.

2. Efficient Feature Projection: While our pipelines may each support providing thousands of features, the average model may use 20-100 features for inference. Hamilton lets us dynamically prune the DAG to run only the nodes needed to produce the features requested by a model. This reduces our overall latency, especially when coupled with Hamilton’s async integration which lets us only fetch from data sources that are needed for the request. 

3. Write Once, Run Anywhere: As mentioned, a key concern in feature engineering is to eliminate train-serve skew. Hamilton lets us write transformations for on-demand inference which may have a single record at a time. We can then reuse the same transformation code, swap out how data is loaded (e.g., Snowflake instead of a Redis cache used on-demand), and run large-scale backfills to generate training data. We can orchestrate our Python code anywhere without needing a JVM environment or any other specialized infrastructure. 

4. Compatibility with Our Existing Tools: Hamilton integrates well with our current machine learning ecosystem, which primarily includes pandas and Snowpark. This made adopting it a smoother process compared to other platforms that require complete infrastructure overhauls.

Building the Platform Around Hamilton

We built our entire platform around Hamilton, which is responsible for calculating features in four contexts:

1. On-Demand Feature Calculation: When models require features for individual records, the system can calculate these features in real time. We execute Hamilton DAGs as Ray tasks within a FastAPI server running on Kubernetes. 

2. Batch Processing: Many of our features, especially lifetime or long-windowed aggregations (e.g., last two years), can be calculated in batch. We use Hamilton to help modularize pipelines for both pandas and Snowpark. These features are then loaded into our offline feature store (Snowflake) for training and batch inference, but they can also be cached in our online store (DynamoDB) for online inference.

3. Stream Processing: While we haven’t fully integrated stream processing yet, we plan to use Hamilton to help write dataframe transformations that could be used with a streaming database like Rising Wave, thanks to Hamilton’s integration with Ibis. These streaming features would be available in both the online and offline stores for inference and training. 

4. Backfills: We support backfills across the three main execution contexts, and spin up ephemeral Kubernetes jobs to handle the processing (ideally pushing down the execution to Snowflake). Hamilton lets us use reuse the same code to backfill, but also helps structure our DAGs to perform temporally-accurate backfills to compute features as-of a specific point in time, which could be unique for each entity in the selected population (i.e., millions of unique observation times to compute).

By using Hamilton as the core execution engine of our platform, we’ve been able to maintain a high degree of flexibility while supporting the complex requirements of feature engineering at scale. Pipelines that used to take months to build can now be onboarded in days or weeks. We’ve also reduced silos across departments and created a centralized place to engineer new features. 

Looking Forward

The decision to build our own feature engineering platform was not made lightly, but it has proven to be the right choice for us so far. With Hamilton, we’ve created a system that:

• Scales efficiently with our data

• Is flexible enough to adapt to changing business needs

• Ensures consistency between training and production environments

And most importantly, we’ve built a platform that gives us complete control, allowing us to continue innovating to meet the growing demands of our machine learning initiatives, all with the goal of better serving our customers.  

Some of our future roadmap items include evaluating the companion self-hostable UI that Hamilton comes with to cover:

• Observability - making sure we catch and find data issues quickly will be key to ensuring reliability of the platform. With Hamilton we have feature provenance by default, and can inject data observability easily. Hamilton has features here that we’re looking to leverage.

• Catalog - a UI to search for, find, and understand features. We expect that as our platform grows, curation & understanding of the feature platform system will become more important, e.g. data dependencies, governance, cost, etc. We’re looking to invest in this area as time goes.

Resources

• https://github.com/dagworks-inc/hamilton

• https://hamilton.dagworks.io/en/latest/

• https://www.tryhamilton.dev/

Agentic Design Patterns

LLMs have people’s heads spinning. The sheer freedom afforded by an API call is intoxicating – not only can you now *talk* to a computer, but you can use that computer to execute (and even write) code for you!

This has led to an influx of Burr users, many of whom want help modeling their systems. After building a successful chatbot, they inevitably migrate towards building agents. While the word agent is buzzy, we define it as follows: an agent is anything that uses the output of an LLM to do something on a users behalf (rather than just parroting it back to the user).

Although agents are generally seen as new-fangled and chaotic, we’ve noticed the emergence of a few common patterns that enable reliable usage of them in a production setting. In this series of posts, we will explore those – walk through basic dos/don’t's of agent development, and show how to implement them with Burr!

In this post we’ll be talking about how to use “tool-calling” to build an agent that responds to a query by selecting a “tool” to call from a list of “prebuilt tools”, then “using” it.  We’ll go over some sample code, then talk about how Burr can help bring this system to production. We assume OpenAI as our LLM, but this example is straightforward to extend to other LLMs.

Note: you can see the code for the demo here and run it yourself/copy+paste to get started!

Before we start, let’s just get some grounding around language…

“Tools” vs “functions” (vs “structured outputs”)

It can be somewhat confusing what these mean. Do agents physically use tools? or do they call functions? or? Are they interchangeable? Are they not? What do they actually mean?

TL;DR: a “tool” and a “function” are synonyms.

We think people make this sound more complex than it really is. For the vast majority of purposes, a “tool” maps to a “function”, where function is the software engineering concept of function. That’s because we’re talking agents that are built via code, and so the simplest way to do something on a user’s behalf, is to invoke (i.e. call/ execute) a function. However, since agents are a higher level concept, we’ll also use a higher level concept, i.e. “tool”, to describe what they can do / use. So in short they’re synonyms and is why you’ll hear a phrase like “tool call” instead of “tool use” because it’s merging “function call” with “tool use”.

To make this more confusing, OpenAI recently deprecated its function-calling API in favor of its tool-calling API (really…) while Anthropic’s documentation refers to it as the entirely unambiguous Tool use (function calling) feature… So the industry has yet to align.

To simplify for the rest of this post, we’ll generally use “tool” to describe the general agent pattern. But where appropriate use the term “function” to actually explain what’s going on (we’ll also drop the quotes).

But what are “structured outputs” then?

When asking an LLM to pick or choose a tool to use, what is returned in the response is usually a JSON representation of the name of the tool and what should be provided to the tool to make it work. Because it is JSON versus free text, it is structured. What people have figured out is that you can co-opt this behavior (e.g. OpenAI’s structured outputs) to make the LLM provide a response that is entirely JSON based, without having an actual tool to use. This can help you generalize outside of specific tool-calling APIs.

While we elect to directly use the tool-calling APIs in this post, adding a library like instructor can help abstract this distinction from you. We leave this as an exercise for the reader…

Burr

First, let’s go over the framework. Burr is a lightweight Python library you use to build applications as state machines. You construct your application out of a series of actions (these can be either decorated functions or objects), which declare inputs from state, as well as inputs from the user. These specify custom logic (delegating to any framework), as well as instructions on how to update state. State is immutable, which allows you to inspect it at any given point. Burr handles orchestration, monitoring, persistence, etc…).

You run your Burr actions as part of an application – this allows you to string them together with a series of (optionally) conditional transitions from action to action.

Burr comes with a user-interface that enables monitoring/telemetry, as well as hooks to persist state/execute arbitrary code during execution.

You can visualize this as a flow chart, i.e. graph / state machine:

And monitor it using the local telemetry debugger:

While the above example is a simple illustration, Burr is commonly used for AI assistants (like in this example), RAG applications, and human-in-the-loop AI interfaces. See the repository examples for a (more exhaustive) set of use-cases.

Knowing which tool to use

LLMs have emerged as a way to determine a user’s intent and also carry that intent out. For instance, say you have a query as simple as:

What is the weather in San Francisco?

Aside from hardcoding the response foggy, this used to be a hard problem. Nowadays, this seems fairly easy – you can just ask an LLM!  

State of the art models, however, do not have the capability to answer a question like  this as they are largely closed off from the internet  (although some tooling has begun to bridge that gap). Instead, their superpower is determining the user’s intent and telling you how to figure it out yourself (LLMs can’t run code, but they can tell you what you should do). So, rather than asking the LLM what the weather is in San Francisco and expecting a result, you should ask the LLM how do I determine the weather in San Francisco, given that I can do X, Y, and Z.  X, Y, and Z could be whatever preset tools (i.e. functions) you have. If you’re lucky, X could be “use a simple weather API, given JSON parameters”, and the LLM would know about it – providing the JSON parameters so you can do the call. Providers such as Anthropic and OpenAI have sophisticated APIs to make this easy — they select from a list of tools and generate the parameters.

Zooming in on the “Ask LLM for tool parameters” part of the flow chart above. For the LLM to know that a tool exists, you need to explain the tool to the LLM as part of the context that is sent. Different LLM providers can do this differently, but in general you are going to be doing the following:

Let’s now dive into how you’d set up an agent written in Burr to use an LLM to determine what tool to use and submit a response to the user.

A Straightforward Tool Calling Pattern with Burr

Implementing tool-calling with Burr is just a thin layer on top of any provided LLM API that supports tool/function calling a.k.a. “structured outputs”. 

Let’s generalize our use case slightly and move from answering a specific “what the weather is question” to answering any question in general. To set this up, we need to provide the LLM the set of tools that you use to get an answer with. E.G. you have functions that wrap APIs to query the weather, order coffee, text your wife that you love her, etc… And all you want to do is convert natural language (i.e. user queries) to input parameters to use one of those tools. 

We will also add a fallback tool that allows the LLM to attempt to answer using its own knowledge (you don’t need a tool to know what 10 x 10 is…).

First, let’s define some functions, i.e. tools. For brevity here we only show our weather_tool.

Not shown but we can also have _text_wife_tool, _order_coffee_tool, and any others we want. We will also have a _fallback function that gives out a simple statement in case we can’t answer using the desired tools or the LLM fails to answer otherwise.

Let’s gather these into a prebuilt list. What we’re doing here is simple — we’re formatting function signatures as the OpenAI compatible type; we’re doing a clever trick of using Python’s inspect module to tell us about the function itself. This means that it’s easy to add new functions, i.e. tools, for the LLM to know about. Alternatively, you could use pydantic here to define the tools. Note: there is additional validation of signatures that we have not shown in the code below.

We now call OpenAI to choose the tool. We model this as specific Burr action. The input is state with the user question, and the output will be the details of what tools should be run and with what values. Note the LLM call was finicky — it didn’t always want to choose a tool, and sometimes got lost in the instructions… But, with a little bit of prompt engineering (experimented on with Burr’s UI), we were able to get reasonable behavior:

The next step is to create a Burr action to exercise the chosen tool. We could model this as a single Burr action that would execute whatever the tool call was. Or, we can use a cool trick to make specific Burr actions for each tool, but still keep the code concise. We can do that with Burr’s .bind() feature, that allows you to define a single action with a parameter and turn that into multiple actions, by “binding” different values for that parameter. In this case, we can parameterize the tool-calling function, binding the tool (i.e. function) to create a Burr action for each tool we have defined (skip to the application builder code to see how). Each of these will take in the tool parameters provided by OpenAI, and output the raw response in some raw (dictionary) format:

We could also have this be a single action that chooses between tools (given the tool set in state), but the nice thing about the one-tool-per-node approach is that we can view all the tools in the graph (see below).

Finally, we’ll have an action that formats the results using OpenAI — this just takes the raw result and puts it in a format the user might like. Note you will likely want to restrict this to some structured object to make it friendly for web rendering, e.g. with a tool like instructor. Below we just return the output straight from OpenAI.

Finally, we tie this all together to create the Burr Application. In this case we’re hardcoding tool names for simplicity, but we could easily use a single dictionary throughout this and the above code to make this code DRYer:

This creates a very reasonable looking graph — we loop back to the input when we’re done so the user can ask another question:

Let’s test this out…

And the result is:

The current weather in San Francisco is clear with no cloud cover, a temperature around 24.81°C (76.66°F), and a humidity of approximately 51%. Wind speeds are mild at about 5.13 km/h, with gusts up to 6.31 km/h. Visibility is excellent at 16 km. There is no precipitation expected, and the UV index is at 0, indicating no immediate health concerns from UV exposure.

Yeah, not foggy. Hmm, not sure I trust this whole AI thing…

Is Burr more than I need?

While the API calls are simple, and the code above could be written without Burr, the orchestration + capabilities Burr provides add significant benefits to both development and production.

Modeling with Extensibility

As everything is represented by a graph, the workflow is self-documenting. You can view the graph, read through the application builder and get a good sense of how your LLM is thinking. Furthermore, as you inevitably add more tools (and adjust prompts), you have the guarantee that changes are located in specific Burr actions, and compare code across runs.

Furthermore, should you want to add additional graph-level concerns (error conditions, etc…) you can do so simply by adding actions & edges without touching any of your other code.

Monitoring + Visibility

In addition to ensuring your graph is more readable, Burr comes with a one-line change to add observability that is paired with an open-source user interface. This allows you to debug as you progress. With tool-calling (LLMs can be fickle and occasionally have a tough time matching intent with the tool to use), this is debugging insight is very useful. Should the LLM not give what you desire, you can:

1. Examine the state

2. Look through the opentelemetry data for chat completions

3. Recreate with specific test-cases to iterate and fix the issue

Should your actual tool, i.e. function call, be giving you errors, you can also annotate these tools with the `@trace()` decorator (and even the code inside those functions). What this will do is have those function invocations also logged to the Burr UI!

Additional Production Concerns

Using Burr also makes it easier to ship to/manage your app in production. You can:

1. Add persistence/track sessions with Burr’s persistence API – another one-line change to hook into the database of your choice

2. Deploy in a FastAPI server using Burr’s state-typing capabilities

3. Gather data for evaluation datasets (more documentation on this coming soon…)

Wrapping up/Further Reading

In this post we talked about how to build a simple tool-calling app with Burr + OpenAI.

While this is more of a toy application, we’ve found a lot more profound/powerful agentic patterns that we’ll be talking about in the coming weeks/months. Up next:

1. How to string together a chain of tools to answer a question/carry out an action

2. How to query and call a set tools in parallel (E.G. for research)

3. How to leverage multiple agents to answer a question

4. How to have agents that work together constructively to answer a question/complete a task

And more! We’re always looking for new use-cases, so if you have one you’re curious about, reach out and we can show you how it would work with Burr!

In the meanwhile, 

• Join our Discord for help or if you have questions!

• Star our Github repository if you like what you see

• Check out the recent OpenTelemetry integration for additional observability

• See the code for this post among our examples

You don’t know you need it, until you need it.

When data teams are small, it’s simple enough to ask the person next to you,  “hey what produced this table prod.customer_orders”, or “what data created this particular predictive churn model?” But as teams grow, and more data is produced, keeping track of how it all relates becomes a challenge. 

In this post, we’ll go over how you can track all the data that you produce with Hamilton & OpenLineage. Specifically we will cover what Hamilton & OpenLineage are, and how Hamilton’s new OpenLineage adapter integrates with the OpenLineage ecosystem.

Subscribe now

What is Hamilton?

Hamilton is a standardized way of building dataflows, a.k.a. “pipelines”, in Python. The core concepts are simple – you write each data transformation step as a single Python function with the following rules:

1. The name of the function corresponds to the output variable it computes.

2. The parameter names (and types) correspond to inputs. These can be either passed-in parameters or names of other upstream functions.

This approach allows you to represent assets in ways that correspond closely to code, naturally self-documenting, and portable across infrastructure.

After writing your functions, you call a driver to execute them – in most cases this is a simple import/run (specifying the assets you want computed and letting the framework do the rest), but it provides options to customize execution:

Hamilton powers a wide range of use cases including feature engineering, ML training/inference pipelines (relevant to this post), RAG pipelines, and more.

Visit tryhamilton.dev for a quick interactive introduction in the browser.

What is OpenLineage?

OpenLineage is a standard for tracking the flow of data (e.g. data lineage) through different tools and systems in an organization. It is a form of observability.

If this is new to you, a shipping tracking system is a good analogy. Imagine you order a product online and you want to know where it is at any moment — from the warehouse to the delivery truck to your doorstep. OpenLineage does something similar, it tracks data as it moves through various stages of processing and transformation within an organization. By having this tracking in place, teams can understand where the data came from, what happened to it, and where it ended up. This helps with:

• troubleshooting issues

• ensuring data quality

• complying with regulations

• and more

It becomes easy to understand the data’s journey, like knowing exactly where your package is and when it will arrive.

The OpenLineage standard

OpenLineage is a standard, meaning it is supported by multiple tools, allowing you to have precise lineage as data move through different components of your system (ETL, machine learning training and inference, predictions, etc.). The standard is implemented as a JSON schema that describe “data events” (read more).

The OpenLineage ecosystem is composed of producers, which are tools that transform data and generate OpenLineage events (e.g., Hamilton, Airflow, dbt, Dagster, Snowflake, Great Expectations, etc.), and consumers which are platforms to observe and catalog data transformations across producers (Amundsen, Marquez, OpenMetadata, etc.).

Who needs lineage?

Everyone wants lineage, but nobody wants to implement it. In sectors where auditing and governance are requirements, OpenLineage is critical to understand data impacts and comply with regulation. However, even when it’s not critical, OpenLineage is can be an enormous boost for productivity because it allows you to monitor what data within your organization is used and plan changes that won’t break someone else’s pipeline. Because lineage is hard to implement, teams avoid extra effort to integrate it, but Hamilton makes it easy for you! In fact, you get a lot of “lineage” with Hamilton, even more than what’s specified in the OpenLineage specification: you get logical lineage and provenance with Hamilton, and then observability with a single line if you’re using the HamiltonTracker with the Hamilton UI. In this post we’re leveraging this to make it easy to emit OpenLineage events.

How Hamilton produces OpenLineage events

To work, OpenLineage needs to know about what data was loaded & saved. When writing your Hamilton code, can generate the required metadata by using the materializer feature. There are several ways to use it:

1. Use the @dataloader and @datasaver decorators to load/save your data and produce metadata (see example below).

2. Materializer decorators with .execute()

3. Materializer classes with either .with_materializers() or .materialize().

Below we show the first approach (1):

import pickle
from typing import Tuple

import pandas as pd

from hamilton.function_modifiers import dataloader, datasaver
from hamilton.io import utils

@dataloader()
def user_dataset(file_ds_path: str) -> Tuple[pd.DataFrame, dict]:
    df = pd.read_csv(file_ds_path)
    return df, utils.get_file_and_dataframe_metadata(file_ds_path, df)

...

@datasaver()
def saved_file(fit_model: ModelObject, file_path: str) -> dict:
    with open(file_path, "wb") as f:
        pickle.dump(fit_model, f)
    return utils.get_file_metadata(file_path)

Importantly, you don’t need to import anything related to OpenLineage when defining your data transformations. Your code remains platform agnostic, as shown in the above example.

To integrate with OpenLineage, we use Hamilton’s very extensible lifecycle API system. The lifecycle API allows you to quickly write custom “adapters” that can observe things before and after graph & node execution. The OpenLineage adapter will produce lineage events as you execute your dataflow. To enable this feature, simply add the adapter to your Hamilton Driver:

import my_dataflow
from openlineage.client import OpenLineageClient

# write to HTTP, e.g. marquez; there are others like Kafka, File, etc.
client = OpenLineageClient(url="http://localhost:5000")

# create the adapter
adapter = OpenLineageAdapter(client, "my_namespace", "my_job_name")

# add to Hamilton
# import your pipeline code
dr = (
  driver.Builder()
  .with_modules(my_dataflow)
  .with_adapters(adapter) # <-- add the adapter here
  .build()
)
# execute as normal -- and openlineage events will be emitted
dr.execute(...)

In terms of OpenLineage specifics, currently Hamilton emits the following facets:

1. SourceCodeJobFacet - with the entire source code of your Hamilton dataflow.

2. JobTypeJobFacet - specifying a BATCH job.

3. A custom HamiltonFacet - that specifies the parameterization of the Hamilton dataflow.

4. A StorageDatasetFacet & DatasourceDatasetFacet - for files loaded & saved

5. A SQLJobFacet - if SQL is detected as being used.

6. A ErrorMessageRunFacet - if your Hamilton dataflow execution fails.

If you need more facets, please open up an issue / open up a PR to contribute!

Example code

If you’re after some sample code to get started, we’ve got an example here to help you get started.

Use case: Boost Airflow pipeline observability with Hamilton

If you’re using Airflow’s Python & Kubernetes operator — you can’t emit lineage events from arbitrary Python code. But if it’s running Hamilton(!), you can now easily make sure that all your data, ML, and AI workflows provide lineage and visibility over what’s happening.

A sketch of the code would be the following:

1. Put Hamilton within the Python / Kubernetes task.

2. Add the OpenLineage adapter when constructing the Driver.

3. Execute as normal — and gain OpenLineage insights everywhere!

@task
def my_python_task(some_argument):
    """This is a function that will run within Airflow"""
    from openlineage.client import OpenLineageClient

    # write to HTTP, e.g. marquez
    client = OpenLineageClient(url="http://localhost:5000")

    # create the adapter
    adapter = OpenLineageAdapter(client, "my_namespace", "my_job_name")

    # add to Hamilton
    # import your pipeline code
    dr = (driver
      .Builder()
      .with_modules(YOUR_MODULES)
      .with_adapters(adapter) # <-- add the adapter here
      .build()
    )
    # execute as normal -- and openlineage events will be emitted
    dr.execute(...)

Does the Hamilton UI consume OpenLineage events?

Not yet; an integration is on the roadmap.

Subscribe now

Summary

This post introduced Hamilton, OpenLineage, and how they can power your data lineage efforts. Hamilton helps fill observability and lineage gaps in Python code via its graph structure and OpenLineage event emission.

Hamilton is now an OpenLineage producer, and the Hamilton UI should become an OpenLineage consumer soon! This will provide the single pane of glass connecting data, ML, and AI observability.

We Want to Hear from You

If you’re excited by any of this, or have strong opinions, drop by our Slack channel / or leave some comments here! Some resources to get you help:

📣 join our community on Slack — we’re more than happy to help answer questions you might have or get you started.

⭐️ Hamilton on GitHub

📝 leave us an issue if you find something

Other Hamilton posts you might be interested in:

• Hamilton + Airflow (GitHub repo)
  

However, the blogs constitute a fragmented ecosystem (blogging platforms, personal websites, company blogs, etc.) which can make it difficult to browse content or find posts you liked.

To tackle these problems, we’ll build a personal knowledge base from our favorite blogs and use retrieval augmented generation (RAG) to search through content using a chat interface. We’ll use a combination of open-source Python tools to progress towards a production-ready application!

Outline

1. Ingest blog articles with dlt and LanceDB

2. Build a RAG application with Burr and LanceDB

3. Monitor a RAG application with OpenTelemetry and Burr UI

Follow along via the GitHub repository

1. Ingest blog articles

While blogs are scattered across the web, most of them adhere to the RSS (or the similar Atom) standard format for publishing and subscribing to content updates. The RSS specifications define required and optional fields to describe the channel and its entries (i.e., the blog website and its articles). Note that even if the fields are standard, how they’re used and the information they contain can vary greatly between feeds.

Finding the RSS feed

Even though not many people interact with it directly, an RSS feed is available on most blogging sites, even personal ones built with popular frameworks! If there’s no visible RSS icon or link, you can find its URL with the browser’s developer tools. Hit F12, then under the Elements tab, use CTRL + F to access the search bar and look up rss, atom, and feed to try to the RSS link.

When you access that link, you’ll find a not-so-human-friendly XML page. If you’re lucky, it will have some formatting and interactivity, but it’s not always the case.

Retrieving RSS entries using Python

Luckily, no need to manually parse XML because the feedparser library takes care of it! The documentation is also an awesome resource to learn more about the RSS specifications.

All you need to do is to make an HTTP request to the RSS feed and give the response to the feed parser.

Then, you can use this object to access information about the RSS channel and entries

For this tutorial, we’ll be using the Substack RSS feed available at my_substack_url.com/feed because each entry’s field content contains the full HTML page of the blog article. We can actually see it in the <content:encoded> tag from the browser.

For blog sites that don’t include the full HTML, you will have to use the entry’s link attribute to make another HTTP GET request and retrieve the HTML page.

Storing RSS entries in a database

Now, we need to decide how to store blogs. The library dlt, the “data load tool”, makes it trivial to convert a Python dictionary into a table schema and store objects in a database.

First, we use the @dlt.resource decorator to create a Resource that gets an RSS feed, iterates over available entries, and yield a dictionary per entry.

To ingest data into a LanceDB database, all we need to do is create a Pipeline object with the destination "lancedb" and set a pipeline name. Then, executing the pipeline by passing the rss_entries to the .run() method will create the database and start populating it.

Simply change the URL given to the function rss_entries(rss_feed_url=) in order to ingest other blogs!

Text processing for RAG

So far, we stored large HTML blobs. To create a good RAG application, we also need to clean the text and create right-sized chunks to provide meaningful information to the LLM. For this purpose, we create simple functions cleanup_html(), split_text(), and contextualize(). They respectively remove tags from an HTML string, split the text on punctuation (. ! ?), and uses a rolling window to join chunks into larger windows. Having some overlap between chunks is helpful during retrieval.

Note that you can use any text processing library for this step (unstructured, LlamaIndex, LangChain, etc.)

To apply these transformations, we create a simple function text_chunks() with the @dlt.transformer. It takes in an RSS entry produced by the rss_entries resource and yields “contextualized” chunks from the entry.

Compared to the earlier code, we use the pipe operator | to indicate that the data from rss_entries should flow into the text_chunks transform step.

Embedding text and storing vectors

An embedding is a vector representation of a text string. For RAG, we want to store each text chunk with its vector representation to enable vector search (also called semantic or similarity search). When the user asks a question our application, we’ll embed its query (i.e., create a vector), search for nearby vectors, and retrieve their associated text chunk.

For this purpose, we’ll use the vector database LanceDB. One key feature is the ability to delegate it the responsibility to compute embeddings. This simplifies RAG applications because it avoids having to write code to generate vectors and move them around. Other benefits will become apparent in the section 2. Build a RAG application. Also, the LanceDB + dlt integration means we can embed our text chunks at ingestion with a single line of code.

Compared to the previous snippets, we give the rss_entries(...) | text_chunks object to the lancedb_adapter an specify to embed the field text.

Note that LanceDB + dlt will require a few lines of configurations to specify which embedding provider and model to use, and the necessary API keys.

Next steps: Scheduling data ingestion

Next step would be to deploy and schedule the dlt pipeline to run every day. While this sound daunting, dlt has a straightforward walkthrough in their documentation. Also, the support for incremental loading means it can automatically avoid ingesting the same document multiple time and wasting API calls for embeddings, which can be costly at scale. We leave this exercise to the reader!

2. Build a RAG application

“Retrieval augmented generation” is a set of techniques rather than a specific algorithm. Burr is a framework to write the logic of RAG applications, and LLM agents more generally, as a graph. By using the concepts of action and state, the behavior of complex agents can always be visualized and debugging becomes easier.

Burr also solves many challenges to productionize agents including monitoring, storing interactions, streaming, and more, and comes with a rich open-source UI for observability (see 3. Monitor a RAG application).

Writing RAG logic

For this tutorial, we’ll write a simple chatbot that receives a user query, retrieves relevant blog chunks, and returns an LLM-generated answer.

To define actions, we use the @action decorator, which specifies the values it reads from and writes to the State. Also, the function must take a State as the first argument and return a State object.

The action relevant_chunk_retrieval() takes in a text input user_query, searches for most relevant chunks, and updates the state with the retrieved chunks and user query. Notice how simple the LanceDB code is. The embedding for the user’s query is automatically computed and used for similarity search.

The second action bot_turn() uses the OpenAI Python client to send a chat completion request to gpt-4o-mini. The key element of RAG is including the relevant chunks of information in the prompt sent to the LLM. Using Burr makes very clear how data moves through the application and the observability features allow to inspect in detail each step (see 3. Monitor a RAG application).

Assembling the RAG application

To build a Burr Application, you need to pass it actions and define valid transitions as tuples (from, to). The application must also define an entrypoint from where to begin execution. In this case, the RAG chatbot can only go from relevant_chunk_retrieval -> bot_turn and bot_turn -> relevant_chunk_retrieval. Finally, calling the .build() method on the ApplicationBuilder() will return the Application object.

Launching the RAG application

Since a Burr Application is a simple Python object, you can launch and try it from a notebook or a script during development one of these options:

• .run(...) will execute the application until it hits a specific condition

• .iterate(...) allows you to execute it in a for loop where you can manually handle conditions

• .step(...) will execute a single action and wait for further instructions (great for debugging)

Note there exists async alternatives .arun(), .aiterate(), .astep()

Below, we launch the application with .run() and make it halt after the action bot_turn. We must provide a dictionary of inputs that includes a value for user_query since it’s required by the relevant_chunk_retrieval action.

This will complete one loop user_query -> relevant_chunk_retrieval -> bot_turn and return a state object that contains the chat_history with the chatbot’s reply.

Now, converting this to an “open-ended” conversation is trivial! All you need is a while loop to call application.run(...) multiple times. Make sure to have a mechanism to exit the loop (😅) and to provide a new user_query on each iteration (here, we get a new input at the end of the loop)

Next steps: deploying Burr as a web service

As you’ve seen, writing an application with Burr is a matter of defining action and state. It makes developing reliable RAG and LLM agents with predictable and debuggable behaviors much simpler.

Yet, Burr provides also provides a lot of value when it comes to moving your application to production. Via the ApplicationBuilder, it is possible to add many functionalities:

• track execution and gather telemetry (which we cover next)

• persist the state, allowing to resume later

• hooks to send alert and trigger other systems

A next step for the reader would be to deploy the Burr Application as a web service accessible to other users. Here’s a guide on this topic from the official Burr documentation.

3. Monitor a RAG application

This simple RAG application already involves multiple components (agent, vector store, LLM, user interface) and this number could grow (multiple LLMs, caching, routers, web searches). An observable RAG system should help you answer questions about:

• User behavior patterns (frequent topics, difficult queries)

• Application performance (guardrails, LLM randomness, user ratings)

• System performance (latency, throughput, resource usage, API calls)

• System errors (bugs from code or the interaction between code and data)

Instrumentation with OpenTelemetry

OpenTelemetry is an open-source cross-language tool that allows to instrument, generate, collect, and export telemetry data (metrics, logs, traces), and constitute an industry-recognized standard (learn more).

Once you enable OpenTelemetry, you need to instrument your code, which means patching your code to produce and report telemetry. Then, auto-instrumentation refers to adding a few lines of code to fully instrument a library. The Python open-source community implemented auto-instrumentation for many popular libraries.

For example, add these 2 lines of code to instrument the requests library:

OpenLLMetry is an open-source Python library that automatically instruments with OpenTelemetry components of your LLM stack including LLM providers (OpenAI, Anthropic, HuggingFace, Cohere, etc.), vector databases (Weaviate, Qdrant, Chroma, etc.), and frameworks (Burr, Haystack, LangChain, LlamaIndex). Concretely, it means you automatically get detailed traces of API calls, retrieval operations, or text transformations for example.

Monitor your application with Burr UI

OpenTelemetry is a middleware, meaning it generates data, but doesn’t provide a storage or dashboard. When using Burr, it can automatically track and store OpenTelemetry data for you, but you can also set up your own OpenTelemetry destination.

To do so, simply add the clause .with_tracker(use_otel_tracing=True) to the ApplicationBuilder. Below, we also instrument the lancedb and the openai libraries used by our RAG application.

Afterwards, launching and using the Application will generate telemetry for Burr to track. Then, using the command burr in your terminal will launch the Burr UI!

Importantly, it allows us to observe our RAG application and precisely track the user requests, the retrieved chunks, the generated responses, etc. The Burr UI is continually improving and has many more features, including:

• monitor execution in real time, see it move through states

• track attributes such as token counts, latency,

• inspect code code failures

• create test fixtures from specific states with one click

Conclusion

This blog showed how to kickstart an observable Retrieval-Augmented Generation application over blog articles. Building towards production-readiness, Sections 1. and 2. include next steps to schedule data ingestion and deploy your RAG application via a web service!

More Resources

• Join our Discord for help or if you have questions!

• Github repository for Burr (give us a star if you like what you see!)

• OpenTelemetry

• Example + README

First, however, let’s go over the basics of Hamilton.

Hamilton

Hamilton is a lightweight python library for building dataflows (any computation requiring data) out of python functions. It can run wherever Python does, and is widely used for anything from simple statistical modeling (this post) to more complex ML, to online RAG in web services and more complex ingestion pipelines for AI systems.

The core concept is simple – you write each data transformation step as a single Python function, with the following rules:

1. The name of the function corresponds to the output variable it computes.

2. The parameter names (and types) correspond to inputs. These can be either passed-in parameters or names of other upstream functions.

This approach allows you to represent your data in ways that correspond closely to code, are naturally self-documenting, and portable across infrastructure.

After writing your functions (assets), you write a driver that executes them – in most cases this is a simple import/run (specifying the assets you want computed and letting the framework do the rest), but it provides options to customize execution:

Hamilton has a notebook integration that we’ll be using in our post, allowing you to define your modules in a cell and reference them in drivers later. You can do this with the %%cell_to_module command, which will update a variable with a module pointer and plot the dataflow defined by the module.

Once you define your module, you will have access to the module pointer as a variable with the name of the module, allowing you to build the driver, just as we did above. See this post for more information on how the integration works.

Ecosystem Additions

Hypster + Hypernodes

recently open-sourced his library hypster — a lightweight configuration system for interacting with, iterating on, and managing AI/ML workflows. Together with Hypernodes, these form a frontend of sorts for Hamilton — enabling quick iteration + narrowing the dev → prod gap. While it is independent of Hamilton, it draws from the same concepts and can be used to configure/manage Hamilton DAGs. Hypernodes, while still in early days, allows you to use Hamilton DAGs within Hypster configurations. He recently talked at the Hamilton community meetup about his work — sharing how he used these tools to build and iterate on a powerful, extensible RAG Q/A system. Have a listen — there’s a lot to learn! He’s really paving the way on leveraging hierarchical DAGs (hamilton DAGs as black-box steps in his pipeline) to lower cognitive burden in developing complex workflows.

Gilad writes about the philosophy behind these libraries and more in his recent post 5 Pillars for a Hyper-Optimized AI Workflow.

Gilad’s Substack

5 Pillars for a Hyper-Optimized AI Workflow

Intro In the last decade, I carried with me a deep question in the back of my mind in every project I've worked on: “How (the hell) am I supposed to structure and develop my AI & ML projects…

Read more

2 years ago · 3 likes · Gilad Rubin

Flower Power

Volker Lorrmann released FlowerPower, an lightweight orchestration library built to run Hamilton workflows! It leverages APScheduler to run Hamilton workflows, allowing you to configure Hamilton workflows and launch/run them on a cadence! Together with the Hamilton UI you can now run, track, and iterate on your pipelines in a distributed setting.

Wren AI

Wren AI is leveraging Hamilton + Async mode to power their core product. You can read their original post on how they used Hamilton to scale through 1500+ concurrent requests here, and our co-post with them on new async features + async upgrades:

We’re really excited about this — while there are a lot of new text-to-SQL tools out there, Wren’s OS implementation is rising above the rest!

DSP Decision Engine

Capitec open-sourced their decision engine library, spockflow — meant to make automating decision trees easy and traceable. They built it using Hamilton for traceability, creating both custom decorators *and* wrapping Hamilton execution/visibility.

Specifying and executing decision engines is easy with the @Tree.condition decorator, which dynamically constructs a Hamilton DAG representing a decision tree:

You can get started using the library by visiting their repository.

NaturF Model

Researchers out of Oak Ridge National Laboratory and Pacific Northwest National Laboratory published the NaturF (Neighborhood Adaptive Tissues for Urban Resilience Futures) model in the Journal of Open Source Software (JOSS). This is a 132 building parameter model that uses building topology to measure the impact of existing/proposed developments on urban microclimates. NaturF produces parameters the the WRF (weather research forecast) model can ingest.

Under the hood, NaturF leverages Hamilton to compute the parameters — this makes it easier to visualize, trace/debug, and recompute specific portions of the DAG. You can see the graph that produces parameters here — it’s part of the paper!

You can read through the documentation here. NaturF also uses GeoPandas to run geographic computation in addition to the geopandas extension for Hamilton.

And more…

Along with the all the exiting libraries outlined above, we are constantly accepting user contributions, promoting blog posts, and sharing out new libraries — reach out if you have/want to build on top of Hamilton!

We want to hear from you!

If you’re excited by any of this, or have strong opinions, drop by our Slack channel / leave some comments here! Some resources to help you get started:

📣 join our Hamilton community on Slack — need help with Hamilton? Ask here.

⭐ give Hamilton a star on github

📝 open an issue if you find any bugs/want new features

We recently launched Burr to create LLM agents and applications.

First, let’s quickly go over Burr.

Burr

Burr is a lightweight Python library you use to build applications as stateful graphs. You construct your application out of a series of actions (these can be either decorated functions or objects), which declare inputs from state, as well as inputs from the user. These specify custom logic (delegating to any framework), as well as instructions on how to update state. State is immutable, which allows you to inspect it at any given point. Burr handles orchestration, monitoring, persistence, etc.

You run your Burr actions as part of an application – this allows you to string them together with a series of (optionally) conditional transitions from action to action.

You can visualize this as a flow chart, i.e. graph / state machine:

Burr is often (but not always) used to build interactive LLM applications. State management, visibility, and persistence is all critical with AI, and Burr is meant to ensure that you get those capabilities for free and not have to think about them. To aid in visibility, Burr allows you to log spans in an OpenTelemetry-compatible format as well as any arbitrary attributes during execution.

Burr comes with a user-interface that enables monitoring/telemetry that we will showcase in this post. To illustrate, we’ll be borrowing from one of our favorite examples: a multi-modal chatbot. It emulates something like the ChatGPT UI, using a model to check the safety of the response, a model to decide the mode, and a model for each response type (image, code, question-answer, etc…).

You can run this with the following command (note you must have an OpenAI API Key set as OPENAI_API_KEY):

pip install "burr[start]"
burr # opens on port 7241

Then navigate to “demos” on the left hand side, and click on chatbot (or streaming-chatbot, they illustrate the same concepts). Preload the existing chats, or follow the instructions to run your own.

You can watch it run through when you submit (note, it will terminate early on check_openai_key if you don’t have one), and preloading any of the demo runs will have the full set of visibility available.

Traces/Spans

Burr functions as a lightweight OpenTelemetry provider (see this post for more details) — it can both produce and ingest spans/traces (if they occur within the execution of a burr node).

To visualize, Burr presents a full waterfall view with timing

You can expand all (or individual) actions to view spans, by clicking on the (+) sign next to Action:

You can view the corresponding state at any given point in time as well, as well as a host of other debugging information (code, graph shape, test case fixture creation, etc…):

This helps you isolate issues post-hoc — determining the exact point in code at which, say, a hallucination occurred, or, if code breaks, the exact state that caused the issue.

Attributes

Burr allows you to go one step deeper — you can log any number of attributes and attach it to a trace/action. This works especially well with OpenTelemetry auto-instrumentation (see this post for instructions), and you can use Burr’s Trace API to log any serializable data.

Attributes show up inline, allowing you to see the prompt, the chat history, etc…

Note this will show up anywhere an instrumented call is made, so for every LLM call you can view attributes. You can also view the number of tokens (prompt/completion), as well as other metadata about the chat itself.

This helps you dig into the exact context in which any step ran, allowing you insight below the function-level.

Insights

Burr also enables you to look at the application as a whole — answering questions on total token usage, broken up by step. The insights tab gives you application-scoped data aggregated by step/span, enabling you to determine which steps cost the most to answer.

Note we currently have a few LLM-specific insights, but plan to add more shortly — including direct cost metrics, timing, and aggregations of other (more arbitrary metrics). Stay tuned!

Future Plans

We’re just getting started. We’re particularly excited about OpenTelemetry data — by automatically instrumenting LLM calls, we actually have enough data to launch an interactive prompt playground, from any point in the application’s call-stack! Here’s a sneak-peak of a streamlit prototype that allows you to load up an LLM call at any point and reproduce/play with it.

We have a host of other ideas in the works — but we operate largely on feedback. If you have any feedback/suggestions, please let us know!

More Resources

• Join our Discord for help or if you have questions!

• Github repository for Burr (give us a star if you like what you see!)

• OpenTelemetry

• Example + README.

Specifically, we usually break our application logic into functions – these functions call other functions, which call other functions, which eventually call out to a model or a vector DB. In order to understand both what happened and why it happened, we need insight into those functions.

This is why we built the `@trace` decorator — it enables us to flip a switch on any function called within a Burr step and gain full visibility into the inputs, the execution time, and the return value of that function.

In this post we will briefly go over Burr, talk about how to use the new @trace decorator, and how to explore the results in the Burr UI. We will then briefly talk about other instrumentation/visualization capabilities.

Note that the version of the Burr UI we show in this post is (as of August 23rd) available as an RC – we will be releasing it properly early next week!

Burr

Burr is a lightweight Python library you use to build applications as stateful graphs. You construct your application out of a series of actions (these can be either decorated functions or objects), which declare inputs from state, as well as inputs from the user. These specify custom logic (delegating to any framework), as well as instructions on how to update state. State is immutable, which allows you to inspect it at any given point. Burr handles orchestration, monitoring, persistence, etc.

You run your Burr actions as part of an application – this allows you to string them together with a series of (optionally) conditional transitions from action to action.

Burr comes with a user-interface that enables monitoring/telemetry, as well as hooks to persist state/execute arbitrary code during execution.

You can visualize this as a flow chart, i.e. graph / state machine:

You can use the local telemetry debugger to view apps (which, as you will see, also provides some OpenTel integrations):

A Simple Agentic Chatbot

Let’s graduate from the hello world example and build something useful — a simple agentic chatbot that handles multiple response modes. It will be agentic as it makes multiple LLM calls:

1. A call to determine the model to query. Our model will have a few “modes” — generate a poem, answer a question, etc…

2. A call to the actual model (in this case prompt + model combination)

With the OpenAI API this is more of a toy example — their models are impressive jacks of all trades. That said, this pattern of tool delegation shows up in a wide variety of AI applications, and this example can be extrapolated cleanly. You will note this is a similar example to our prior visibility post — you can skim through if you’re on a visibility spree and just came from that one!

To leverage Burr, we model our application as a graph of actions. The basic flow of logic looks like this:

Let’s take a look at the chat_response function that powers the nodes answer_question and generate_code, depending on the value of prepend_prompt:

You will note that this uses the helper function _query_openai, which can easily be adjusted to query any LLM. We will be instrumenting this in a bit.

The graph creation + instantiation looks similar to what we showed above (a few pieces left out for brevity). Note we’re reusing some functions (chat_response), by binding different function parameters to create different nodes in the graph.

Gaining Visibility

We can explore how this looks in the UI. The following is from two prompts in a row

As you can see, we have some access to data, input, etc.. through the view, but we don’t have the internals of how the prompt was manipulated.

Using the @trace decorator

With the @trace decorator we can augment the data above. Specifically, we’re going to surround two of our functions with @trace to gain insight into how they work. The _query_openai method because it is useful, and the _get_openai_client method for fun.

We can view the spans (toggled globally by the +/- button near the action header, or the +/- button near individual actions), we can see that a set of spans have been logged that correspond to a set of attributes (see the inline database icon):

Clicking the database icon to the right of a span opens up the attributes logged and gives us insight into the parameters (prompt, model, chat_history), as well as the return value (labeled return):

Not only can we see the full prompt (see `prompt` attribute), but we can also tell that we have an additional colon (`::`), and that it took 3.345 seconds to respond. We also have some simple debugging information on the function that was instrumented and the hash of the code. This response glosses over the treason, name change, failed marriages, and financial challenges that plagued him later in life.

Further Instrumentation

That said, this is just a toy example. The true power of AI does more than provide answers a simple wikipedia article would show, and Burr can be used for a lot more than just a chatbot.

The @trace decorator makes this feasible – instead of just gaining visibility at the Burr level (state, results), you can dig in as deep as you want.

In addition (and in complement) to the trace decorator, you can auto-instrument most of the popular (vendor and OS) model providers & vector databases, as well as leverage any other instrumentation. You can view all this data in the Burr UI, in your favorite OTel provider, or both! See our previous writeup for more information.

More Resources

• Join our Discord for help or if you have questions!

• Github repository for Burr (give us a star if you like what you see!)

• OpenTelemetry

• Example + README

One of the most challenging aspects of building Generative AI applications is understanding what happened and why it happened. In addition to the standard monitoring concerns (how long did this step take, what caused the exception the program threw, etc…), we have to deal with a plethora of Generative AI-specific challenges. Did this model get fed a reasonable prompt? Did it produce a reasonable result? If not, why? When you’re building agentic applications (those that leverage AI calls to make automated decisions), this gets even more complicated.

In this post, we’re going to talk about how you can leverage the industry-wide OpenTelemetry standard to instrument Generative AI applications built with Burr and provide visibility into the aforementioned questions. We will present the basics of OpenTelemetry,  walk through a simple application implementation with Burr, and show how you can get visibility over the decisions your “AI/Agent” make in a few lines of code, using your preferred OpenTelemetry vendor along with the builtin Burr UI. We will then show how you can utilize custom instrumentation to gain additional monitoring for Generative AI-specific concerns.

OpenTelemetry

OpenTelemetry, or OTel for short, is a standard that has skyrocketed in popularity in recent years. It is a CNCF incubating project, with an open governance model to ensure no single vendor steers it too much in any specific direction.

It is now supported by a large array of vendors, ensuring that how you instrument is not a cause of vendor lock in, which forces observability companies, i.e. those that ingest OpenTelemetry data, to compete by building better features.

OpenTelemetry defines how applications can export the following information:

1. Traces – a request’s journey as it travels through the application  

2. Metrics – various data points (aggregate or individual) that represent some observation of the application at a given point in time

3. Logs – what the application printed while running

In this post we will be focusing on traces, which OpenTelemetry models with the following concepts:

1. A span – an individual component of execution with a start/end time and exception data attached to it. This can be recursive – spans can have sub-spans, etc…

2. A trace – a group of spans that represent a journey through the application

3. Attributes – data attached to a span to make debugging easier.

OpenTelemetry provides APIs for ingesting this data, as well as client-side tooling that implements this. In python, the opentelemetry-api package defines high-level interfaces, and the opentelemetry-sdk as a default implementation, is provider-agnostic client.

Using OpenTelemetry tracing in python is simple – you just need to get the global tracer and exercise it:

So, OpenTelemetry is popular and easy to use, but why should you consider using it, and how can it help with AI observability?

1. OpenTelemetry is vendor (a.k.a. visibility provider) agnostic, meaning that you can easily switch between vendors. In this case we’ll be using Traceloop to demonstrate, but you can easily get started on any of these.

2. OpenTelemetry is openly governed, meaning that you can dig in to understand the decisions they made and even be a part of it!

3. Data visibility is the first step to understanding the questions of why a model made the decision it did — OpenTelemetry provides this with tracing and attributes (logging the input/output/other model data).

4. There are a host of extensions available to OpenTelemetry — in this case we will be using Traceloop’s open source openllmetry library to automatically instrument our LLM calls as it supports many LLM & vector DB client SDKs.

Burr

Burr is a lightweight Python library you use to build applications as stateful graphs. You construct your application out of a series of actions (these can be either decorated functions or objects), which declare inputs from state, as well as inputs from the user. These specify custom logic (delegating to any framework), as well as instructions on how to update state. State is immutable, which allows you to inspect it at any given point. Burr handles orchestration, monitoring, persistence, etc…).

You run your Burr actions as part of an application – this allows you to string them together with a series of (optionally) conditional transitions from action to action.

Burr comes with a user-interface that enables monitoring/telemetry, as well as hooks to persist state/execute arbitrary code during execution.

You can visualize this as a flow chart, i.e. graph / state machine:

You can use the local telemetry debugger to view apps (which, as you will see, also provides some OpenTel integrations):

A Simple Agentic Chatbot

Let’s graduate from the hello world example and build something useful — a simple agentic chatbot that handles multiple response modes. It will be agentic as it makes multiple LLM calls:

1. A call to determine the model to query. Our model will have a few “modes” — generate a poem, answer a question, etc…

2. A call to the actual model (in this case prompt + model combination)

With the OpenAI API this is more of a toy example — their models are impressive jacks of all trades. That said, this pattern of tool delegation shows up in a wide variety of AI applications, and this example can be extrapolated cleanly. You will note this is a similar example to our prior streaming post. 

Modeling our Application

To leverage Burr, we model our application as a graph of actions. The basic flow of logic looks like this:

Let’s take a look at the decide_mode function – this is a simple call out to OpenAI (one of a few in the stack).

As you can see, we are directly using the OpenAI API, and parsing the result (note that instructor or OpenAI structured outputs would likely be a better choice to force the mode to produce  what we want, blog post coming soon!). 

The graph creation + instantiation looks similar to what we showed above (a few pieces left out for brevity). Note we’re reusing some functions (chat_response), by binding different function parameters to create different nodes in the graph.

Manual Instrumentation with OpenTelemetry

We have two ways of manually logging custom OpenTelemetry traces with Burr:

1. The Burr tracing SDK

2. The OpenTelemetry SDK

We’ll show the Burr API (1), but you can use the OpenTelemetry SDK interchangeably — it will all get logged to the same place. To add custom traces, we can do something like this manually:

Then to run, we add a simple hook. This ensures that framework calls (E.G. to app.run()...) are traced as well. 

Generative AI-specific OpenTelemetry

OpenTelemetry defines a set of attributes specifically for tracking AI applications. This enables us avoid the manual instrumentation we did above, and instead get it for free!

These take the form of semantic conventions that give some very simple model-related parameters. Furthermore, Traceloop has implemented a variety of SDKs that automatically instrument common clients. You can access them with Traceloop here (and, if you initialize with traceloop, they get automatically set up).

Thus to automatically instrument most gen AI applications, you simply need to call instrument on the appropriate object:

While this is not required for the basics of monitoring with Burr, the auto-instrumentation libraries follows the semantic conventions and logs attributes that are useful in giving insight to the model, as well as the cost of execution, e.g. how many tokens were sent and how many were generated.

Putting this all together, we can see our spans get populated in Traceloop!

Logging OpenTelemetry Traces to Burr

We can take the same traces we logged above and flip a simple switch to log them via Burr’s tracking hooks — this will log to the local Burr server and show the traces in the context of a Burr execution.

We can see traces in the Burr UI with a similar view, as well as associated attributes on a span level:

Wrapping Up

We showed how to gain visibility by leveraging OpenTelemetry. In addition to providing insight into system performance, OpenTelemetry (through either the Burr UI or a OpenTel vendor) affords us visibility into every aspect of our application – the inputs/outputs of models, and how downstream processes leveraged that. So, circling back to what we wanted to know initially:

1. Did this model get fed a reasonable prompt? 

2. Did it produce a reasonable result? 

3. If not, why? 

(1) and (2) are directly answered by attribute-logging with OpenTelemetry, whereas (3) will always be a bit of a guessing game. That said, knowing the facts (inputs and outputs) is the first step to understanding the why, and Burr has features to help you “replay” / go back in time to quickly debug or try different code paths, providing an integrated development & observability experience.

Lastly, OpenTelemetry traces support crossing service boundaries. So, if you are self-hosting your LLM and really want to get into the nitty-gritty, you can instrument your model service and report traces back to, say, Traceloop or Burr, gaining visibility into the model along with the application.

More Resources

• Join our Discord for help or if you have questions!

• Github repository for Burr (give us a star if you like what you see!)

• OpenTelemetry

• Example + README

You can find the code (notebook + modules) for the blog post here.

The Problem

A baby never comes on time. Well, a baby rarely comes on time. In fact, there is a 3% chance that a baby will come on the due date, a 50.5% chance that it will come before, and a 46.5% chance that it will come after.

For parents expecting to deliver naturally, this can be an unsatisfying answer:

• When will my baby be here?

• Well, given that today is July 14, your due date is July 18, and the baby has not yet come, here is the skew-normal probability distribution that describes it (big curve). Oh, and if it doesn’t come by the end of July you’ll want to induce. 

While I am as frustrated about the lack of certainty as anyone (this lesson applies to more than just simple predictions, by the way), I do have one advantage. I’m an absolute nerd – I have a passing fancy for conditional probabilities and pretty plots, as well as an obsession for building elegant data manipulation systems.

In this post, we’re going to calculate due date probabilities using a fairly simple statistical model. In doing so, we will show you how to use Hamilton for production-ready EDA inside a notebook, utilize the builtin calendar library to visualize our results, and reverse engineer the parameters of a probability distribution in an extremely hacky but simple way. All this because the room is painted, the crib is assembled, the go-bag is packed (well, hers is but I’m procrastinating), and really, there is very little else for me to do.

Hamilton

We will go over visualization/modeling tools when we use them, but let’s explain Hamilton a bit before diving in. Hamilton is a lightweight python library for building dataflows (any computation requiring data) out of python functions. It can run wherever Python does, and is widely used for anything from simple statistical modeling (this post) to more complex ML, to online RAG in web services and more complex ingestion pipelines for AI systems.

The core concept is simple – you write each data transformation step as a single Python function, with the following rules:

1. The name of the function corresponds to the output variable it computes.

2. The parameter names (and types) correspond to inputs. These can be either passed-in parameters or names of other upstream functions.

This approach allows you to represent your data in ways that correspond closely to code, are naturally self-documenting, and portable across infrastructure.

After writing your functions (assets), you write a driver that executes them – in most cases this is a simple import/run (specifying the assets you want computed and letting the framework do the rest), but it provides options to customize execution:

Hamilton has a notebook integration that we’ll be using in our post, allowing you to define your modules in a cell and reference them in drivers later. You can do this with the %%cell_to_module command, which will update a variable with a module pointer and plot the dataflow defined by the module.

Once you define your module, you will have access to the module pointer as a variable with the name of the module, allowing you to build the driver, just as we did above. See this post for more information on how the integration works.

Modeling Due Dates

While the underlying probability distribution of delivery is actually very complicated and not well known, according to dataayze (the source for a lot of this post), a skewed-normal distribution is a reasonable representation. The due date is roughly the median, and the left-flank is longer tailed (5th percentile is 4 weeks before the due date, 95th percentile is 2 weeks after). A skewed-normal distribution has three parameters. 

The first two are the same a standard normal distribution, although they have different meanings

• loc – where the curve is shifted (note due to skewness this is *not* the center of mass)

• scale – standard deviation. How wide the curve is.

The final parameter is called “skewness” (a in the code below) – this is how lopsided the curve is. To demonstrate, you can see how the probability density function varies based on the skewness parameter for a simple standard normal distribution centered around 0 with scale of 1.

Once we have the right parameters, we should be able to map the calendar dates to the input and use the probability density function (PDF) to calculate probability of delivery *on* a due-date, as well as the cumulative density function (CDF) to calculate the probability of delivery *before* a due date. Simple, right? Almost – there are two wrinkles:

1. We don’t have the parameters, and they’re weirdly hard to find (in fact, the papers that everyone cites (duration of human singleton pregnancy and gestational length assignment …) don’t represent it as a skewed distribution and instead attempt to estimate the mean/std dev.

2. There are two other factors we need to account for

   1. Given that today is July 14th, the probability of delivering prior to today is zero. So, the probabilities for future dates will change (increase – they all have to sum to one, and we cut out a big chunk)

   2. Gestation is generally not considered safe post 42 weeks – current medical practice will almost always induce after that. 

While (1) would be solved in a better way by finding the data and doing our own analysis (or, asking the author of that website nicely), we’re going to go with maximum effort! We’ll go over how we solve these problems in the code.

Computing with Hamilton

Determining Model Parameters

Recall we didn’t actually know the parameters to the model? Let’s figure them out! We’re going to use the data here: https://datayze.com/labor-probability-chart and reverse engineer what they did (we also reached out for their parameters to compare our results). This part is classic ugly data parsing, that through Hamilton is well organized into functions – we won’t dig too much into the code, but at a high-level, we will:

1. Copy/paste the probability data into our notebook in the raw_data function

2. Compute the raw probabilities, using a lot of funky string parsing (avoiding regexes at all cost)

3. Resample by generating a reservoir of data with 1000 total items (this is clean as the probabilities in the chart had it down to .1% granularity)

4. Fit to scipy.stats.skewnorm()

Why the roundabout manner with sampling? Well, really, it’s not the right way to do things. We should be using curve_fit (or some similar capability), or even analytically solve it. That’s hard, but the scipy.stats.skewnorm.fit() function is easy, and it gets us close enough.

The DAG to compute the parameters looks like this – we’ve broken out scale, a, and loc using @extract_fields.

We can run this code as follows:

Which gives us the following parameters:

{
 "a": -4.186168447183817,
 "scale": 20.670154416450384,
 "loc": 294.44465059093034
}
This yields a pretty intuitive set of density functions:

This yields a pretty intuitive set of density functions:

Again note that while 40 days is the median, the mode is actually around 41 weeks. This also looks close to the data in the website, meaning we’ve reverse-engineered the distribution correctly!

Managing Dates

Before we get around to computing probabilities using the model above, let’s set up a few date-related variables we can use. In this case, we will model the due date from the start date (the first day of the mother’s period), an input to our dataflow. The due date is defined as 40 weeks past the start_date (a 9-month gestation period + a full cycle). 

We restrict our calculation down to just a smaller set of dates (8 weeks before and 4 weeks after), largely to make the data sane. Thus we have possible_dates, which will form the “spine” (common index) for the rest of the data:

This code presents the following (simple) DAG:

Computing Probabilities

Now that we have the base model as well as our dates, let’s actually calculate the probabilities. First we’re going to compute over a larger date range (0->365 days), which we will later truncate. We create a scipy rv_continuous object that allows us to compute the density functions we want. We then do some adjustments to model our initial assumptions – both the current date and the induction date:

This is the meat of the code – we do two adjustments:

1. We truncate to past the current date (if specified), and readjust the probabilities to sum to 1)

2. We truncate after the “induction date” (defaults to two weeks post due-date), and add the entire mass to the induction date

While there are obviously a few simplifying assumptions (fixed induction date, quantized date set, etc…), this is a pretty reasonable model.

Next all we need to do is compute the cdf with a cumsum() operation (note we’re not using the rv_continous cdf() capability as we transformed the probability distribution into a discrete function with different values and cumsum() will work just fine. While we could define our own rv_continous variable, the discrete modeling here is a simple, debuggable approach. We truncate our distribution function series to have the right range, and return them as our final variables:

Finally, we can build the driver and use it to return a dataframe – in this case we’re using the PandasDataFrameResult result builder, which handles joining for us.

Voila! We have our final results.

Analyzing the Results

Let’s take a look at what we produced. Using a simple seaborn plot (code in example, left out for now), we can see the following distributions. 

At this point in the pregnancy, it is quite likely that the baby will come late. The mode (obviously) its induction on august 1st, with a 12% chance of that happening. The median is around the 22nd (a week late).   

The probability distribution above has no name (of which I’m aware, although the shape is suspiciously close to a common birthing position), but it tells an interesting story. And, it changes every day! Rerun each morning to get your new conditional probabilities.

We can also visualize this in a calendar using the builtin calendar library in python, a tool that enables us to format calendars and perform various calendar-related operations. In this case we’re going to be customizing it heavily – subclassing the HTMLCalendar to allow for highlighting of certain days with a scale:

When we run view_date_range on our data, we get the following:

The real reason I’m doing all of this, is, however, because I want to know the probability that my baby shares a birthday with me! 5% is low, but stranger things than 1/20 have happened.

Notes on Workflow, and Future Reading

All this to determine a 5% probability? Well, the end result doesn’t really tell us much (but hey, it’s a probability distribution, so we’re speaking in possibilities and unknowns anyway). Furthermore, this probably could have been a simpler notebook (avoiding the use of Hamilton all together). 

That said, Hamilton made me a lot more productive building this, despite the fact that I was adhering to the rules of a specific framework. I got this productivity gain from the following:

1. I never worried about notebook state – I had visualizations, and it would recompute whatever needs to be (no changing variable names and accidentally attempting to use them)

2. I was able to work iteratively – using the Jupyter notebook extensions I can do the following routine

   1. Compute a variable

   2. Play around with it until I’m happy

   3. Tab that cell in (b) in and surround it with a function, put it into the module

   4. Repeat until I’m happy

3. I get to see the visualization at any point – I can pretty easily figure out what’s happening, what a variable means (that I forgot), etc…

4. I get code out at the end that’s modular, broken up into functions, and easy to navigate!

This was a toy example. In the real world, there would be a lot more analysis, a lot more spot-checking, and a much more complex set of assumptions to navigate. And that’s where Hamilton really shines – at every stage you’ll have a production-ready ETL to work with. With the Jupyter notebook extension you can even use the -w flag with the %%cell_to_module magic to write your module to a file as you’re editing! This enables you to use the notebook as both a notebook and an IDE.

Learn More

If you’re excited by any of this, or have strong opinions, drop by our Slack channel / leave some comments here! Some resources to help you get started:

📣 join our Hamilton community on Slack — need help with Hamilton? Ask here.

⭐ give hamilton a star on github

📝 open an issue if you find any bugs/want new features

We recently launchedBurr to create LLM agents and applications as well.