Monthly Hamilton Meetup and Office Hours
This Tuesday we held our inaugural monthly Hamilton meetup for 2024! We’re writing a quick overview for those of you who missed it (and to entice you to come next time)!
The format of these meetups will (usually) be as follows:
Community spotlight – Hamilton community members showcase and discuss projects they’re working on, their experience migrating/getting started, the problems Hamilton helped solve, and tips for using advanced features
Deep dive – Hamilton maintainers/contributors explore in details a set of Hamilton features or use-cases
Open mic/Roadmap – An opportunity for community members to introduce themselves, chat together, and ask questions (Hamilton-related or not), keeping it topical to data pipelines, MLOps, AI, etc…
Monthly meetups will occur on the third Tuesday of every month at 9:30am PT!
Hamilton office hours will take place on other Tuesdays at the same time. It will be an informal place to ask questions about Hamilton, or help structure/discuss more broadly your ML/LLM efforts.
Meetup
Over 25 people showed up, from a wide range of time-zones! Arthur Andres gave a talk about how Tradewell Technologies leveraged Hamilton to migrate over their feature engineering pipeline. He shared:
Some key concept of feature engineering (using an example of estimating wait time for a taxi ride)
Details about the increasing complexity of managing feature engineering configurations before using Hamilton, especially when trying to introduce new features from development, to test, to production.
How Hamilton helped them organize their codebase and configure different execution modes. They eventually faced challenges when trying to specify feature sets via configurations.
How they leveraged the @resolve decorator for a powerful configuration-driven approach that met their project requirements.
We followed-up with a great discussion on the trade-offs of using @resolve. Other Hamilton users shared their design to solve similar challenges.
Next, we covered a few different challenges, including:
Adopting naming conventions multiple modules and a growing code base
How Hamilton can integrate with PySpark/Polars/PyArrow operations
When to alter the DAG at runtime (based on
final_vars) versus compile-time (based onconfig)
For the deep dive, Stefan presented how to structure a repository of Hamilton code, discussing the trade-offs for team collaboration, maintainability, and scalability. He showcased more advanced function modifiers including @parameterize and @subdag to make DAG components reusable. Finally, Stefan covered the roadmap, highlighting the recently completed items and where Hamilton is headed!
The presentation and roadmap generated many questions, leading to an interactive discussion. This brought up several interesting directions to go next (follow the repository issues for more…).
Thanks to all of you for attending! Our next meetup will be March 19th – Roel Bertens will be speaking on using Hamilton + Spark to build a feature catalog! Recording of last meetup as follows:
Office Hours
On account of the exciting diversity of people leveraging/interested in Hamilton, we have decided to host office hours! Join slack to get access to office hours – they will be Tuesdays at 9:30AM PT (reach out if that timing doesn’t work but you still want to meet).
What’s next
Sign up for our meetups here
Join slack to know about more meetups and office hours!
Watch the meetup recording here
Try hamilton out (in the browser!) at tryhamilton.dev
Check out the OS repo