You might recognize us as the person in your help menu or as the person that just answered your support ticket. Either way, we’re here to help de-mystify our newest feature: The Advanced Data Pipeline!
What is the Advanced Data Pipeline? Why should I use it? When should I use it? These are all great questions that we even asked ourselves when we first heard about the Advanced Data Pipeline. If you’re wondering too then hopefully our takes can help shed some light to this awesome new feature!
I love flow charts, I love the ability to see the big picture. It gives me the chance to not only see all of the smaller pieces but see how they relate to each other. Step 3 might not directly impact Step 7, but I can see how they relate through the steps in between with a nice flow chart. When trying to visualize how certain queries are going to be joined to others when building some pretty complex charts, I used to have to draw it on a notepad to make sure I saw the whole picture. You can ask a few of my colleagues that sit next to me, I have a growing pile of used notepads on my desk. I need to draw, no more. Advanced Data Pipeline gives me the ability to see the whole process my charts are going through. When I have two layers that need to be inner joined in order to remove the rows that don’t fit my criteria, but the results of those need to be left joined to show where some row might not have information from the 3rd layer in order to find the null sets, Advanced Data Pipeline view gives me that visualization.
When an executive is making a business decision based on a process and that process is explained to him or her in a well designed and maintained flow chart, they can see the whole picture. When a Chartio user can be sure that their results are accurate by viewing the whole process the charts are going through, they too can see the whole picture. When the Chartio user knows the results are accurate, the executive can too.
I don’t want to be too bold in saying that I think The Advanced Data Pipeline is the coolest thing about Chartio but it is (at least for me) and also I love metaphors. Think about solving a Rubik’s cube and how you’re constrained by only having the ability to rotate rows. What if you can take it apart by each block and re-attach them? That’s what the Advanced Data Pipeline is to the Basic Pipeline.
Visualizing my thought process is the key to my success when it comes to building charts. Chartio does a great job in helping me do most of the visualizing with the Basic Pipeline but the Advanced Data Pipeline is the complete picture. I started my data journey not knowing anything about databases or SQL. Starting out, the Basic Pipeline was great in helping me build the foundation of learning sql syntax and organizing logic to build queries. As I progressed, I found myself wanting to do more complex data flows but lacking the interface to do this in besides just pure SQL. This is why the Advanced Data Pipeline is great because now I have options! Being able to completely rearrange steps as I envision them, connecting disparate data through multiple types of joins, and the workflow visual are all reasons why I find the Advanced Data Pipeline essential to how I build my charts. Not all charts need to be built using this but it’s great knowing now there is a way if needed.
Rather than embark on a winding road of metaphors and puns that will only put you in a bad mood, I will say the Advanced Data Pipeline has saved my life by allowing me to perform MULTIPLE JOINS. Finally Chartio!!! Joins, joins, all types of joins! Recently, I was assigned a project that required me to search for all customers that fit a specific subset of a subset of characteristics, along with additional information stacked on top in the end for our Finance team. While it would have been delightful to find out that all of our customer information was stored in one location, this was not the case. Anyone working with data knows life is never that easy. In fact, I had to mine through three different data sources to uncover what I needed. I had to merge sets of data on a common ID connecting disparate data through multiple types of joins and filtering on this result set prior to performing a left join with the last data source, the almighty CSV. Without the ability to perform multiple joins, this never would have been possible. Along the way, there were several mistakes made within additional pipeline steps, however the ability to disconnect (temporarily) the selected nodes and reconnect later on in the pipeline saved me minutes, if not hours, in having to completely remake all filter steps and custom columns I had to rearrange in the data pipeline.
The Advanced Data Pipeline truly allows you to transform disparate data with a user interface that matches what is happening behind the scenes. It allows you to diagram data workflows in a way that appeals to both Business and Technical users.
This was a huge success for both the Company and me. After creating the report I quickly became my Dad’s favorite son and became a role model for everyone within Chartio as well. This could happen to you too! Is a possible White House dinner invitation up next? I don’t know. But probably!
What if the only way to reach a conclusion is through a winding road of metaphors? Well, What if you could only add pepperoni to a pizza? Well that’s fine but wouldn’t you want to be able to add as many toppings on that pizza as you like? Well I would! That’s what the advanced pipeline mode does for me. It bakes pizzas!!***. In other words, the basic mode only allows one merge step (boooring!) but the advanced mode enables as many joins as you like!
As time passed and I worked more and more with Chartio, the capabilities I was looking for increased as well. The ability to do multiple complex joins without writing a lick of SQL is a game changer for me. I know some SQL but I’m a far ways from a SQL expert. This kind of thing would have taken me ages to do but now I can accomplish this task in mere seconds. The new pipeline mode really lets me have my pizza and eat it too!
***Does not actually bake pizzas