DAGWorks 2023
A year in review.
2023 was a big year for DAGWorks. We’re writing this post to share our learnings, brag about our accomplishments, and broadcast our vision and goals for 2024 and beyond into the world! While it has been a total whirlwind, we are proud of our accomplishments and excited about what the future holds.
Accomplishments
In 2023, we…
Raised initial capital to support Hamilton and DAGWorks development
Started and finished the YC W23 batch, and the StartX S23 Batch
Forked the old Hamilton repository into dagworks-inc/hamilton
Gained 1100 stars for the new repository
Landed on the front page of hackernews at least 3 times
Were shared on the TLDR newsletter to over 1m people
Released 58 distinct versions of Hamilton — 29 minor and 29 patch — that added many new features and improvements to Hamilton.
Implemented the DAGWorks UI and SDK and released for general availability
Engaged in multiple design partnerships and pilot programs, ranging from small consultancies to larger enterprises
Published 23 blog posts on blog.dagworks.io, which were viewed over 15000 times by 11000 unique users. We also released a select few on TDS.
Built tryhamilton.dev to make onboarding to Hamilton easier
Grew the membership on our Hamilton slack community by 125%
Launched the hamilton dataflow hub, allowing anyone to download and contribute custom Hamilton DAGs
Spoke at multiple conferences and meetups including PyCon Lithuania, PyData Seattle, PyData Global, Data Council Austin, MLOps World, LLMAvalanche, h2o GenAI Summit, MDSFest, SciPy, AICamp San Jose, AICamp San Francisco, Manchester PyData, Feature Store Summit, Budapest ML Forum, BAYPIGgies…
Presented Hamilton to internal audiences at places like Adobe, Apple, IBM, The Federal Reserve Board (first external speakers!), C.H. Robinson, Twin Health, etc.
Acquired our first paying customers!
And this is just a sampling — Hamilton is now a much more sophisticated, powerful, and generally applicable library with a user base that dwarfs anything it has had in the past! We have observed it in use by teams in small companies to large, research/government laboratories big and small, to multi-national enterprises, where it has consistently been shown to improve internal pipelines (of a wide variety) and consequently increase the return on investment on a data teams’ work.
Learnings
While we produced a great many accomplishments this year, it proved to be a roller coaster nonetheless. We learned a lot over time, including:
How to dive headfirst into the complexity and nuance of enterprise sales
What a large swath of different data teams do to get their models to production
A variety of tools/approaches required to build a scalable multi-tenant data platform
What types of people who are drawn to Hamilton initially, and how they use it
Frameworks for change management when Hamilton has only just been introduced to a team
Although some of the above will form our competitive moat, we’ve blogged and plan to blog about our learnings to share with the world. We hope for many more in 2024 — after all, what’s the fun in doing a startup if you’re not learning along the way?
Plans and Vision
While we’ve wound our way down our path (with our fair share of distractions), our vision has not waivered. We believe:
Hamilton can and should be used for all “pipelines” involving data, including ML pipelines, feature engineering workflows, LLM/RAG applications, and micro-services. As DBT did for analytics, i.e. standardized how SQL is written, Hamilton is the best tool to standardize all python “pipelines”.
DAGWorks will be the platform that gives enterprise data (Eng., ML, DS) teams a competitive edge, allowing them to leverage the standardization Hamilton has to offer and get several enterprise products integrated in a one-line code change.
We recognize that the vision above requires a lot of work — our sleeves are already rolled and we’re already back at it. So, let’s end this post with some new-years resolutions. In 2024, DAGWorks will be the year of “show”:
Showcase at least 5 enterprise case studies detailing the transition to Hamilton + DAGWorks and value created.
Show how the DAGWorks platform along with Hamilton can hook into every possible MLOps & LLMOps tool out there.
Show Hamilton to the world — get the repository to 3000 stars and 750 people on slack.
And finally, some thanks
To our open-source community (users and contributors) who push the limit and help us set Hamilton’s direction.
To our customers who invest in our relationship and trust us with their architecture and systems.
To our investors who believed in us from the beginning, and continually offer us sage advice.
To our (small but mighty) team that works tirelessly on aligning, promoting, and realizing our shared vision.
To our family who supports us through the insanity of running a startup.
So, all that said, happy new year, and godspeed! We wish you to have as productive and exciting a year as what we have planned…
Amazing work! Grateful to be part of this project -- encourage you all to try their framework, it's brilliant!