{"id":1984,"date":"2022-12-28T13:35:31","date_gmt":"2022-12-28T11:35:31","guid":{"rendered":"http:\/\/52.91.248.125\/whats-in-the-box\/"},"modified":"2023-02-28T11:54:16","modified_gmt":"2023-02-28T09:54:16","slug":"whats-in-the-box","status":"publish","type":"post","link":"https:\/\/www.orfium.com\/news\/whats-in-the-box\/","title":{"rendered":"What\u2019s in the box?"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Black box testing for non-data engineers with DBT<\/h2>\n\n\n\n

Black box testing is a software testing method in which the functionalities of software applications are tested without having knowledge of internal code structure, implementation details, and internal paths. Let\u2019s borrow that term and use the same analogy to test our black boxes, meaning our dbt models.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

So, adapting the lexicon from software engineering, we have:<\/p>\n\n\n\n

Black Box<\/em><\/strong>: the dbt model that plays the role of transformation<\/p>\n\n\n\n

Input(s)<\/em><\/strong>: the different tables that are used in the query. In dbt we call these sources<\/em> or references<\/em><\/p>\n\n\n\n

Output<\/em><\/strong>: the table formed after the dbt model has transformed the data<\/p>\n\n\n\n

DBT and testing<\/h2>\n\n\n\n

DBT is a data build tool. We use it, due to its simplicity, to perform transformation in our snowflake data cloud for analytical purposes. Not to overstate the matter, but we love it.<\/p>\n\n\n\n

What, exactly, is dbt?<\/a><\/p>\n\n\n\n

Building a Mature Analytics Workflow<\/a><\/p>\n\n\n\n

DBT already offers dbt tests, and performing them is a great way to test the data on your tables. By default, the available tests are unique<\/strong>, not_null<\/strong>, accepted_values<\/strong>, relationship<\/strong>. We can even create custom tests, and there are a variety of extensions out there that stretch dbt functionality with additional tests, such as great expectations<\/a> and dbt-utils<\/a>. These kinds of tests examine the values of your tables, and they are a great way to identify any critical data quality issues. DBT tests look at the output. However, what we want to do is to test the black box, the transformation.<\/p>\n\n\n\n

TDD and Data<\/h2>\n\n\n\n

Working with Large Tables<\/strong><\/h3>\n\n\n\n

More often than not, the tables that we need to build models upon are huge, and accessing billion of rows and performing transformation upon them takes a long time. A 30 minute transformation might be acceptable when it is a part of a production pipeline, but having to wait for half an hour to develop and test the correctness of your transformation is, well, less than ideal. <\/p>\n\n\n\n

Of course you are going to run it against the table, but minimizing the number of runs makes everyone happy. This also limits your Snowflake Warehouse Usage which can save cost and make accountants happy as well.<\/p>\n\n\n\n

Edge cases not covered in actual data<\/strong><\/h3>\n\n\n\n

Another problem we often face is having a dbt model that works for all intents and purposes for multiple months, only to later discover that there are cases which we didn\u2019t think of. Unsurprisingly, having billions of rows of data means that all the possible scenarios are not at all easy to cover. If only there was a way to test for those cases, as well. The solution we at Orfium use is to generate mock data. They may not be real, but they work well enough to cover our edge cases and future-proof our dbt instances.<\/p>\n\n\n\n

Good Tests VS Bad Tests<\/strong><\/h3>\n\n\n\n

Writing tests for the sake of writing them is worse than not writing them at all. There, we said it. <\/p>\n\n\n\n

Let’s face it, how many times do we introduce tests on a piece of software, get excited and, thanks to the quick TDD process, we just gleam with self-confidence? Before you know it, we\u2019re writing tests that have no value at all and inventing a fantastic metric called coverage. Coverage is important but not as a single metric. It is only a first indication and should not be used as a goal in itself. Good tests<\/strong> are the ones that provide value. Bad tests<\/strong>, on the other hand, only add to the debt and the maintenance. Remember, tests are a means to an end. To what end? Writing robust code.<\/p>\n\n\n\n

Tests as a requirements gathering tool<\/strong><\/h3>\n\n\n\n

How many times have we found ourselves sitting in a room with a stakeholder who provides information about a new report that they need. We start formulating questions, and after some back and forth, sooner or later we are reaching the final requirements of the report. So, happily enough after the meeting, we go to our favorite warehouse, only to discover some flaw in the original request that we didn’t think of when we did our requirements gathering. Working in an agile environment that\u2019s no issue. We just schedule a follow-up meeting and reach a consensus for the edge cases. Final delivery is reached. However, wouldn’t it be better if actual cases could be drafted in that first meeting? Business and engineering minds often don\u2019t mesh well, so we can use all the help we can get.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

Establishing actual scenarios of how a table could look like and what the result would be, helps a lot in the process of gathering requirements.<\/p>\n\n\n\n

Consider the following imaginary scenario:<\/p>\n\n\n\n

Stakeholder:<\/p>\n\n\n\n

For our table that contains our daily revenue for all the videos, I would like a monthly summary revenue per video for advertisement category.<\/p>\n\n\n\n

Engineer (gotcha):<\/p>\n\n\n\n

<\/p>\n\n\n\n

1select\n\n2 video_id,\n\n3 year(date_rev) as year,\n\n4 month(date_rev) as month,\n\n5 sum(revenue) revenue\n\n6from\n\n7 fct_videos_rev\n\n8where\n\n9 category = 'advertisement'\n\n10group by \n\n11 video_id,\n\n12 year(date_rev),\n\n13 month(date_rev)\n\n14<\/code><\/pre>\n\n\n\n
Stakeholder<\/p>\n\n\n\n
I would also like to see how many records the summation was comprised of.<\/p>\n\n\n\n
Engineer (gotcha):<\/p>\n\n\n\n
1select\n\n2 video_id,\n\n3 year(date_rev) as year,\n\n4 month(date_rev) as month,\n\n5 sum(revenue) revenue,\n\n6 count(*) counts\n\n7from\n\n8 fct_videos_rev\n\n9where\n\n10 category = 'advertisement'\n\n11group by \n\n12 video_id,\n\n13 year(date_rev),\n\n14 month(date_rev)\n\n15<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
Stakeholder<\/p>\n\n\n\n
That can\u2019t be right. Why so many counts?<\/p>\n\n\n\n
Engineer<\/p>\n\n\n\n
There are many rows with zero revenues, I see. You don\u2019t want them to count towards your total count, is that right?<\/p>\n\n\n\n
Stakeholder<\/p>\n\n\n\n
Yes.<\/p>\n\n\n\n
Engineer (gotcha):<\/p>\n\n\n\n
1select\n\n2 video_id,\n\n3 year(date_rev) as year,\n\n4 month(date_rev) as month,\n\n5 sum(revenue) revenue,\n\n6 count(*) counts\n\n7from\n\n8 fct_videos_rev\n\n9where\n\n10 category = 'advertisement'\n\n11 and revenue > 0\n\n12group by \n\n13 video_id,\n\n14 year(date_rev),\n\n15 month(date_rev)\n\n16<\/code><\/pre>\n\n\n\n
Of course, this is an exaggerated example. However, imagine if the same dialog went a different way.<\/p>\n\n\n\n
Stakeholder:<\/p>\n\n\n\n
For our table that contains our daily revenue for all the videos, I would like a monthly summary on a monthly basis per video for the advertisement category.<\/p>\n\n\n\n
Engineer:<\/p>\n\n\n\n
If table has the form:<\/p>\n\n\n\n
video_id<\/td>date_rev<\/td>category<\/td>revenue<\/td><\/tr>
video_a<\/td>2022-02-12<\/td>advertisement<\/td>10<\/td><\/tr>
video_a<\/td>2022-02-12<\/td>advertisement<\/td>0<\/td><\/tr>
video_a<\/td>2022-03-12<\/td>subscription<\/td>15<\/td><\/tr>
video_a<\/td>2022-03-12<\/td>advertisement<\/td>1<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Is the result you want like the following?<\/p>\n\n\n\n
video_id<\/td>year<\/td>month<\/td>revenue<\/td><\/tr>
video_a<\/td>2022<\/td>02<\/td>10<\/td><\/tr>
video_a<\/td>2022<\/td>03<\/td>1<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Stakeholder<\/p>\n\n\n\n
I would also like to see how many records the summation was comprised of.<\/p>\n\n\n\n
So the result you want it to be like:<\/p>\n\n\n\n
video_id<\/td>year<\/td>month<\/td>revenue<\/td>counts<\/td><\/tr>
video_a<\/td>2022<\/td>02<\/td>10<\/td>2<\/td><\/tr>
video_a<\/td>2022<\/td>03<\/td>1<\/td>1<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Stakeholder<\/p>\n\n\n\n
Why does the first row have 2 counts?<\/p>\n\n\n\n
Engineer<\/p>\n\n\n\n
There are two with zero revenues, I see. You don\u2019t you want them to count towards your total count, is that right?<\/p>\n\n\n\n
Stakeholder<\/p>\n\n\n\n
Yes.<\/p>\n\n\n\n
Engineer (gotcha):<\/p>\n\n\n\n
video_id<\/td>year<\/td>month<\/td>revenue<\/td>counts<\/td><\/tr>
video_a<\/td>2022<\/td>02<\/td>10<\/td>1<\/td><\/tr>
video_a<\/td>2022<\/td>03<\/td>1<\/td>1<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
And all that, without having to write a single line of code. Not that an engineer is afraid to write SQL queries. But really, a lot of time is lost in translating business requirements into SQL queries. They are never that<\/em> simple and they are almost<\/em> never<\/em> correct at first try either.<\/p>\n\n\n\n
Tests so software engineers can get onboard in SQL<\/strong><\/h3>\n\n\n\n
Orfium is a company which, at the time of writing this post, consists of more than 150 engineers. Only 6 of those are data engineers. That might sound strange, given that we are a data-heavy company dealing with billions of rows of data on a monthly basis. So, a new initiative has emerged called data-mesh. This is a program which we practice on a daily basis and are super proud of. One consequence of data mesh is that there are multiple teams handling their own instance of dbt. But, this will be discussed in detail in another post. Stay tuned!<\/p>\n\n\n\n
For the most part, software engineers are not familiar with writing complex SQL queries. That\u2019s not their fault, due to the variety of ORM tools available. However, something that software engineers do<\/em> know how to do very well is to write tests.<\/p>\n\n\n\n
In order to bridge that gap, practicing test-driven development on writing SQL is something that can help a lot of engineers to get onboard.<\/p>\n\n\n\n
Let the fun begin<\/h3>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
We designed a way to test dbt models (the black box). Our main drivers are:<\/p>\n\n\n\n