Data wrangling —cleaning, transforming and enriching data— is the starting point that decides whether a generative AI project in banking succeeds or fails. The "garbage in, garbage out" rule applies: poorly prepared data leads to hallucinations and incomplete answers from conversational agents.
Literally, the expression data wrangling refers to the act of corralling or, in a way, "taming" data so it can be used efficiently for a specific purpose. This is a key activity we must carry out when tackling a project that involves generative artificial intelligence, and on its own it can define its success or failure. It should not be underestimated in terms of time and scope. Why is data wrangling important? The quality of the data we use to train artificial intelligence models is decisive for the quality of the results. You may have read the phrase "garbage in, garbage out" somewhere, which basically captures this idea. This applies both to foundation models and LLMs, which invest billions of dollars in this activity before and during model training, and to specific applications where we use our own data to feed and give context to virtual agents that must solve highly specific tasks within our organization. When implementing generative AI solutions in banking, this activity becomes essential to ensure that interactions are as accurate and safe as possible. In this other blog post on how a conversational AI is trained in banking , we detail how data plays a critical role at every phase of developing these systems. What does doing data wrangling actually mean? While technically this activity involves several steps and technical requirements that apply more or less depending on the company's business, the intended use of the data and the starting point, we can boil the concept down to three main points: Cleaning: correct errors, remove duplicate data and inconsistencies. Transformation: convert data from one format to another so it can be consumed correctly. Enrichment: complement the current data with additional data that improves quality and provides the context the solution needs. In implementing a project involving generative AI applied to a concrete use case —such as customer service or selling a banking product like a credit card or a loan— this process will be one of the starting points for evaluation and work. The state of the inputs the solution will use is a key factor in estimating the scope and duration of the project. If we want, for example, our conversational agent to interact with customers by answering complex questions about the bank card's discounts and benefits in effect at different stores, we will need that information to be in the right format, understandable to the solution, and constantly up to date. What data formats exist? When we talk about data formats, we refer to the way certain data is stored and ready to be used or consumed. In this sense, there is a basic distinction, as follows: Structured data: data that is in a predefined structure with defined types and labels, for example, tables in Excel or CSV files, databases, etc. Unstructured data: data or information that comes from less predefined formats such as text files, documents, emails, PDFs, images or even videos. Beyond the fact that each format has its particularities and is more optimal for storing and serving different types of data, as a general rule for using them to feed generative agents, there are general considerations we must keep in mind. Mainly, we must understand that generative models will perform better when data is structured in a way that makes it easy to locate, understand and consume. Just like for people, when we receive properly structured documents —for example, with a table of contents, appropriate use of text hierarchies, labels, specific data formats like numbers, currencies, percentages, etc.— our interpretation is much simpler, faster and more accurate. Additionally, we must keep in mind that humans have a superior capacity for inference and common sense that lets us deduce or fill in missing information from context. As for generative models, while they may try, at least for now, they do not have the right capacity to do so. This can lead to hallucinations or incomplete answers, depending on how much freedom of action the model has. What benefits does correct data wrangling bring to solutions based on conversational agents? As we mentioned at the start, a solution that does not properly treat the data provided as its knowledge source has a high chance of not fulfilling its purpose correctly. But strong final performance is not the only benefit we can expect from applying good practices here. Some additional benefits can be the following: More consistent answers, with lower risk of hallucinations and incomplete information. Easier maintenance and management of knowledge bases. Faster information processing. More agile error detection and correction. Shorter project implementation times. Final thoughts Generative artificial intelligence is a disruptive technology that came to change the rules of the game across every industry, with banking being one of those with the greatest potential to achieve gains in efficiency and customer experience. Applying conversational agents is one of the most obvious use cases to harness its potential, but they require context and rigorous data handling. As noted above, data wrangling is a key starting point, sometimes underestimated. We must approach it with the importance it deserves, since it can define the success or failure of the implementation even before the first interaction with end users begins.
What is data wrangling and why does it matter in banking? Data wrangling is the process of cleaning, transforming and enriching data so it can be used efficiently. In banking it defines whether a generative AI project succeeds: garbage in, garbage out.
What are the three key steps of data wrangling? Cleaning (fix errors, remove duplicates and inconsistencies), transformation (convert data into a consumable format) and enrichment (add data that brings context and quality to the solution).
What benefits does good data wrangling bring to conversational agents? More consistent answers with lower risk of hallucinations, knowledge bases that are easier to maintain, faster processing, more agile error correction and shorter implementation times.