Odenräd: Hackathons at Oden

We continued our semi-annual tradition, despite the Covid-induced disruption, and held our fifth Odenräd in end-October 2021. This is our hackathon, where engineers and non-engineers take a day long break from their work to explore fun, novel, and sometimes wild ideas to express their creativity and collaborate across teams. In this post, I’d like to layout our Odenräd process, the unique challenges (and opportunities) we encountered this year with a hybrid work model, and highlight some of the really neat ideas that our teams came up with.

At Oden, we take our Norse mythology seriously, and grab every opportunity to bastardize it to suit our purpose. The name Odenräd was derived by putting together the Swedish word räd (i.e. raid) with Oden to indicate the frantic, exploratory, and somewhat plundering nature of this process.

The Odenräd Process

We typically schedule our Odenräds towards the end of each half-year, to try to take advantage of our Odenräd ideas during planning for the next quarter. While it is not expected that Odenräd projects are designed towards specific business problems, there still are several that end up making an outsize aspect on our product, our customers, and our teams. We start by soliciting ideas for projects - each Oden-ite is encouraged to submit theirs. These ideas are collected and made visible to all, allowing people to decide which project they would like to take on, and individuals self-organize into teams based on shared project interests.

Once teams are formed, teams can choose to do some exploratory prep work before the scheduled start. We start at noon on a Wednesday where we assemble and each team briefly outlines their idea. This allows any late changes to teams or projects. Teams then disperse and are free to proceed with their projects. Teams have until Thursday 1 PM (a total of slightly over 24 hours) to complete their projects, when all of us regroup, and present/demo our ideas. Teams are free to use whatever tools they need, but we do encourage each team to create a project-specific public Slack channel to capture their brainstorming/discussions etc.

We provide dinner, breakfast, and lunch to allow teams to work uninterrupted on their implementations, although teams decide how/when they work (more about this in the remote versus in-person discussion later). Following final presentations/demos, we vote for projects in two categories: "Most Likely to Impact Oden" and "Wow Factor", and provide winning teams with small prizes, besides of course bragging rights.

This Odenräd was our first in a post-Covid hybrid work setting - with a mix of in-person and remote team members, spanning multiple timezones (-7 to +8 UTC). While this posed a few challenges, this also demonstrated to us our ability to collaborate really effectively, even remotely. Physical challenges of not being able to whiteboard/spitball ideas in person, or sharing a meal with a teammate were partly overcome with online collaboration tools, some amount of pre-Odenräd exploration and prep, and doing remote team meals over Zoom. What was interesting to see was how each team figured out a mix of these that worked best for them, and team member surveys indicate that the level of collaboration was at par with previous mostly in-person Odenräds. Teams also figured out ways to leverage timezone differences to maximize productive periods without extreme sleep-deprivation.

Odenräd Projects

We are a data-rich company. We collect all sorts of really interesting data from factories that represent properties of complex physical-mechanical-chemical activities, giant machines, material, people, products, quality and their choreography within the manufacturing process. We also capture information about our users, their usage patterns, their preferences, and about the business of our customers and potential customers. In addition, we have very diverse teams, often spanning multiple disciplines within engineering, along with finance, marketing and sales. The combination of this data, coupled with our team diversity makes for a highly fertile playground for Odenräd projects.

We now highlight a few projects from four categories: I. Novel ways to gather and visualize data II. Using ML and AI to infer/predict information III. Distributed and optimized data processing IV. Fun Projects

I. Novel Data Collection and Visualization

Huginn: One of Oden's (data gathering) ravens

Factories are notorious for connectivity problems. This project integrated metrics from our ingestion pipeline to provide live visibility of connectivity of each factory - including input data rates/valid/invalid data etc. Customers can then choose to be alerted based on these metrics. A view is shown below:

Cute HMIs

Data we collect is generated from large physically distributed machines, and this project attempted to overlay the collected time series data onto the physical distribution of the data sources that generate them.

II. ML/AI based inference

What's being produced by this line?

Metadata about the process typically comes into Oden via integration into external systems (e.g. a MES system) or from manual data entry. This project leveraged ML to automatically infer the product metadata from process data. This allows us to provide much better real-time visibility, as well as more accurate time alignment for any further analysis. This used a combination of a DNN with a live model scoring framework.

Maxonomy Force 1

It is important for us to contextualize data we collect from the factory floor with knowledge about the process. We do this using our taxonomy, where we map metrics to groups and associate canonical names, units and process relevant labels to them. In this project we built a Word2Vec like model that allows us to capture semantic and syntactic distances between metric names to automatically perform this mapping. We call this Maxonomy - as is allows us to take Taxonomy to the Max. A schematic of the approach is shown below.

Motifinder: Show Me That Pattern

In this project we used time series motifs to define patterns of interest in data, and building a query engine that finds other instances of those. We built this into our Explore engine, that allowed users to click drag and select a pattern of interest, and then have the system automatically find other instances of it. This is very useful for root-cause analysis.

The interface allows users to see which other periods in time had the same patterns in the data, and then drill down to identify the common causes.

III. Distributed and Optimized Data Processing

Natural Language based Data Querying for Non-Experts

In this project, we built templated natural language queries that allow our end-users to use a more natural way to interact with our data. Internally, we converted these queries to BigQuery SQL to run it against the stored data.

Building the Oden Query Language and Use for Real-time Alerts

In this project we built extensions to SQL that allow us to express time-series and interval intersections more effectively for our customer problems. This also allows us to effectively use our timeseries optimized data store Clickhouse. We then demonstrated the use of this for real-time alerting. We extracted rules from our ML models that identify poor performance based on process metrics, translated these into OQL (our new custom Oden Query Language), and ran them against Clickhouse to generate real-time alerts.

Transpiling Calcmetrics

One of the really core functionality we provide our end-users is to apply arbitrary functions (written in Javascript) to streaming data, its instantaneous and windowed values, and its combinations. This is hugely valuable, but is also inordinately expensive to compute in a distributed system where the metric arrival order is not guaranteed, and where there may be gaps in the metric timeseries. This project attempted to transpile our JS functions into the Oden Query Language - allowing us to perform the calculations on read - likely to have a huge impact on our compute costs, and allowing more immediate 'backfilling' of new definitions.

IV. Other Projects

Rube Cube

This was an ambitious project that included building a "manufacturing line" represented by a Rube Goldberg like marble run, attaching sensors to it, connecting them to an Oden device, and ingesting and visualizing the data within Oden. It was an absolute blast that combined hands on assemply, hardware, software, and data visualization.

Sonification

What if we could deliver a complex monitoring experience that is completely passive through the sonification of machine data? This project involved converting the collected multi-variable timeseries data from each line on each factory into an audio stream - to be able to represent the data in ways that we had never considered interacting with. A really neat example clip is included below.

For the clip, we used the following mapping from metric data to sound.

Metric	Sonic
X & Y Diameter	Tonality
X & Y Diameter vs. set limits	Key
Line speed	Tempo
Screw RPM stability	Percussion timing
Melt Pressure stability	Percussion timing

Your Face is Your Passport

We use the Kisi app to control physical entry into our office in New York. This project leveraged deep learning to build a face recognition system that integrated with the Kisi APIs to enable us to unlock our office with our faces.

Oden goes Carbon Neutral through Offsetting

In this project, the team spent time investigating different approaches to offset our carbon footprint - office and travel in addition to personal carbon - selected one, and committed to using it. With our cloud compute relying on GCP which is also carbon neutral, this effectively made each employee at Oden carbon neutral. Not a project that required much development, but one that had a significant impact on us.

So What's Next?

Odenräds are great for us to incubate new ideas, work across many disciplines, have loads of fun, and collaborate. Being remote, in person, and hybrid has not diminished the experience for our teams, and as we can see from this little sampling of projects, we end up so many creative and invaluable set of ideas. In case you're wondering, yes Sonification did win the Wow Factor Award, and Predicting Products did win the "Most Likely to Impact Oden" award. We cannot wait for our next Odenräd!