3 Steps to a Data-Driven Content Quality Approach

Today’s data-driven business world knows one thing well: what doesn’t get measured doesn’t get done. And vice versa: what gets measured, gets managed. So how do we apply those popular truisms to content quality management in a global, multilingual scenario? And what are the common pitfalls to watch out for?

In day-to-day conversations at the workplace, terms like “data”, “metrics”, “information”, and “KPIs” are often used interchangeably. However, understanding the distinctions between concepts that are behind all those terms pays off big time when trying to manage and improve content quality and content performance across multiple teams, companies, languages, and markets. So let’s first set the terminology straight (as we always should whenever working on global content quality), and then see how to apply a data-driven approach to your global content Quality Management practice.

The map of the data-driven world is shaped like a pyramid

One way to interpret the idea of being data-driven is through the model called a “DIKW pyramid” (short for Data – Information – Knowledge – Wisdom):

Data-Information-Knowledge-Wisdom pyramid for data-driven content strategists (from http://libraryguides.lehigh.edu/TRACseminar/understandinginformation)
Being data-driven with your content strategy means understanding what “data” means

The raw signals we gather from the outside world are data. They are meaningless unless context is provided. For example, what does the number 3576 mean for your content quality management program? Hard to tell.

Now, what if I told that 3576 is the number of visitors to your corporate website? That’s more structured and has a specific context. However, it’s still far from being useful if you’re trying to make sense of how your global content performs and whether it’s good or not. So let’s agree that is still data which is trying hard to get to the next level.

OK, taking that one step further still: 3576 unique visitors have signed up for our free product trial in France during the last 4 weeks from the product web page. Now we finally have some information. It gives you a solid basis from which to draw further insights. For example, by asking questions like “How does that number compare across geographies?”.

So now you take the top 4 of your target markets with their respective languages – Spanish (Argentina), English (United States), German (Austria), French (France) – and compare this information for each of them. You notice that the first two locales (both based in the Americas) have brought in a larger absolute amount of trial users, while the second two locales (both based in Europe) have a higher % increase of trial users month-over-month, despite having lower absolute numbers. At this point, we have detailed knowledge about that particular situation.

Finally, you realize that you don’t have to do anything at all about this difference between Americas conversion rates and Europe conversion rates – because it is, in fact, not statistically significant. As you also know that all other regions and languages have smaller traffic when compared to the top 4, you decide to stop tracking this metric entirely for the next 6 months and focus on something more useful instead. That’s best described as wisdom.

Note: In the business world, an applied version of the DIK(W) pyramid is data – metrics – indicators. Here, metrics roughly correspond to information, and indicators roughly correspond to knowledge. The most important 1-3 indicators for a business area are often called Key Performance Indicators, or KPI for short. Why do we want to select and prioritize just a small handful of indicators? Because each measurement has its own cost, and because it helps us avoid “death by dashboards”.

Leading metrics and lagging metrics

Suppose we have knowledge about how well our multilingual digital content matches our pre-defined requirements. In other words, we’ve defined indicators of our content quality. Those indicators can be based on various models for atomistic and holistic quality measurement, or even a combination of such models. We actually can find out the value of quality indicators before we make our content public and expose it to our audiences. In fact, that’s what most companies do as part of their Quality Management strategy for digital globalization programs.

We also have knowledge about the business impact that the very same multilingual content has achieved after being published and read by our end users. In other words, we’ve defined indicators of our content performance. In the world of digital globalization, those indicators can be based on various web content marketing metrics (for example, bounce rates, clickthrough rates, and conversion rates). However, we can only find the values of those indicators post factum: once the global content has been pushed out to the big wide world, there’s no turning back.

Now comes the big question: how do those two indicators, content quality and content performance, relate to each other? For example, does a better quality score for Spanish (Argentina) localized email content always come with an increase in the clickthrough rates for those email campaigns? In other words, does content quality predict content performance?

If yes, we say that our content quality indicator is the leading metric. Content performance indicator then becomes the lagging metric. We’re using the values for the leading metric (quality) to get a notion of what the lagging metric (performance) would likely be in future.

Word of caution, though: correlation doesn’t always equal causation. The fact of content performance always going up (or down) together with content quality does not yet mean that one is the direct result of the other. There might be other independent factors that influence both quality AND performance (just like with ice cream sales and deaths by drowning). So finding out that, for your multilingual content, quality and performance are indeed correlated is just the first step on the long way to discovering the real nature of this intricate, yet strategically important, relationship.

Capture and analyze all data on quality, not just pieces of it

So you might ask: how do I make this journey shorter? How do I get to the bottom of what’s influencing my content performance and understand whether content quality is indeed the culprit? That becomes especially hard if you can’t read & understand most of the languages that your team localizes and publishes your global content in.

Unfortunately, there is no universal answer to this question. However, one useful piece of advice is to approach content quality from a holistic perspective. Focusing on just one aspect of multilingual content quality (e.g. only the translation quality, or only internal review feedback, or only human expert judgment) and ignoring everything else is highly hazardous because this is NOT how your end users and readers will perceive your content in the real world.

Instead, try to get the whole 360-degree picture by capturing the entire range of sources from which you already get, or can get, any data on quality of your multilingual content. This gives you a better chance of spotting any lurking variables affecting the quality-performance relationship. Here are some ways to do this:

  • If your global content is a software app and you’re localizing the user experience (and the UI in particular), blend the software testing results with linguistic quality inspection results.
  • If you’re producing technical content that gets translated into several languages, combine the source language quality measurement with the target language quality measurements.
  • If you’re crowdsourcing translations for your customer support portal, merge your senior translators’ or language moderators’ feedback with your end user translation quality ratings (e.g. 1-5 stars).
  • If you’re applying Machine Translation for your user-generated content, combine automatic metrics (including quality estimation) with human assessment.
  • If you’re doing a third-party evaluation of localized content that was done by another Language Service Provider, juggle your editor’s review results with the output from automatic translation QA tools.
  • If you’re using in-country reviewers to revise & approve your multilingual copy, make sure you’re capturing every single piece of their feedback (even if it had been sent through a text message, in the middle of the night, to the mobile phone of your boss that has been offline at the time). Then compare their feedback to sentiment analysis that captures what your customers say.

How do you currently compare your content quality measurements with your content performance metrics? What are some of the results that you’ve recently seen? Does quality correlate with performance, or do each of those live their separate lives? Are there other variables besides quality that influence content performance? Share your experience in the comments section!

Content Quality Management starts from requirements

Dictionaries are an early form of content requirementsWhen thinking about content quality management and performance of global content, many experts tend to focus exclusively on evaluation, measurement, or assessment of quality. However, there is so much more to Content Quality Management and Content Quality Assurance than the act of checking alone! 

Many of the processes essential for delivering high-impact, high-quality content in multiple languages actually have to happen before AND after any quality checks. If these processes are not in place, your organization might be wasting time-to-market and budgets on sub-optimal content and inefficient, costly QA practices. Here are 6 steps to avoid that, inspired by Six Sigma DMAIC approach:

  1. Define and share requirements
  2. (Produce content – author and/or localize)
  3. Measure quality
  4. Analyze results
  5. Improve both content AND requirements
  6. Control and repeat

Today, we’ll be focusing on the first item: establishing requirements for your content before you produce it so that you can tell “good” from “bad”, and sharing them across the entire global content supply chain. We’ll cover the rest in future blog posts, so stay tuned.

1. Define and share requirements

Why spend time on defining what “good quality” means for your content?

Most heated arguments about quality usually happen when people have very different pictures in their minds of what “good” quality is. To avoid this blunder ourselves, let’s first consider a few definitions of quality:

  • How good or bad something is” (Merriam-Webster)
  • “The standard of something as measured against other things of a similar kind; the degree of excellence of something” (Oxford Dictionaries)
  • “The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs” (ISO)

The key insight to take away from those definitions is that quality is always relativeWe can only have a meaningful conversation about the quality of a work product in question (a piece of content, for instance) by comparing it with something else: pre-existing norms, requirements, standards, rules, examples, metrics, or even past experiences.

So agreeing on requirements for content is paramount to even start hoping to achieve quality. The problem, however, is that these requirements are so often communicated implicitly that we don’t even pause to think about it. Everyone surely knows what type of content I feel will work best for our audience, readers, and users. Right?

Wrong. As your content production team gets larger than 1 person, you’re in for a big surprise. By the way, that happens much sooner than you may realize – just imagine a VP or another corporate stakeholder making edits to content that contradict your “common sense”, or a freelance writer you hire to produce a blog post for you, only to discover you end up with something totally unusable, off-brand.

By the time you are localizing – even if it’s just into 2-3 languages – the amount of people on your extended content team that need to know what “good content” means to your org will have grown by a factor of 10. If you’re the one responsible for content quality management and for driving global content performance in your organization, it’s YOUR job to keep all of these people on the same page. And you have to do it every time, regardless of their location or company. Otherwise, consistency will always remain an elusive goal.

How to explicitly define requirements in content quality management

We’ve already discussed that each piece of content is created for a purpose. Communicating this purpose to the entire team is a good start for achieving quality. However, that alone is usually not enough. What else do we need?

Over the decades, the content industry has crystallized two very powerful ways to define, store, and share content requirements: Style Guides (or manuals of style) and Terminology Databases (or term bases). They are similar in the sense that both are a collection of different rules, instructions, and examples of how to create content (in one language or in several) that will be considered “good” by an organization in a given context (e.g. a specific content type, or a particular project).

In the world of Globalization, Internationalization, Localization, and Translation (GILT), Translation Memories (or TMs) are often used for capturing “good” content for future reference and reuse, and thus can be considered a specialized form of content requirements. Another form of requirements in localization is instructions inside Localization Kits (or LocKits), which usually focus on technical specs essential to delivering “good” localized content in a software app or technical documentation to its end users.

Instead of going into details of how those sources of requirements work, let’s rather consider what they typically consist of. Here’s one way to categorize those content requirements:

  1. Formal wording rules
    • This type of rule prescripts a specific choice of words or sentences in narrowly defined contexts: “When talking about this, always phrase it like this”. For example:
      • Corporate brand terminology and rules for its usage
      • Approved and/or standardized technical terms
      • Canned phrases (e.g. for forcing content reuse)
      • Examples of “good” and “bad” sentences
      • Specific spelling of words to be used
      • Prohibited or banned words/terms to be avoided
      • Usage of abbreviations
    • Formal wording rules typically need human expertise to be validated when doing quality checks. However, sometimes automated methods may be called upon to assist or improve efficiency.
  2. Formal technical rules
    • This type of rule mandates how to present and format your words and sentences. For example:
      • Use of hyphenation, capitalization, and punctuation
      • Measurement units, date and number formats, addresses and phones
      • Character restrictions (e.g. accented/Unicode symbols, control characters)
      • Cross-references style and data sources, both internal and external
      • Length limitations, including sentence length and word length in specific usage contexts
      • Readability levels
    • Formal technical rules can often be validated automatically as part of quality evaluation.
  3. Informal rules
    • This type of rule provides high-level, sometimes even vague ideas on how to best craft text, but is difficult to define in algorithmic terms and often leaves room for subjectivity when interpreted. For example:
      • General expectations for accuracy and fluency, as well as priorities between the two
      • Recommendations on tone of voice, such as avoiding passive constructs or using personal pronouns
      • Style suggestions, such as being concise, clear, compelling, and credible in your headlines
      • Audience profile aspects, such as “13-year old adopted girls that prefer exciting language and use of teen slang”
    • Informal rules almost always require human know-how to be validated during quality measurement. Due to inherent subjectivity, they also often need a person with authority to confirm the final judgment if arguments arise in the quality inspection process.

Too many vs. too few: some requirements are still implicit, and that’s OK

When you’re just starting out to document your content requirements, finding the right balance between capturing too many and capturing too few is key. After all, if you go all the way down to listing basic spelling and grammar rules for each of your languages, your list of requirements will be as long as a good book (or, rather, a stack of those). And, even with software assisting you during the quality evaluation stage, it’s a huge effort to continuously maintain a large requirements database, train new team members (across organization boundaries) on each of those requirements, ensure that requirements are well adapted to each of the languages, and work through inevitable false positives that arise with any automatic validation process.

So how do we keep our content requirements lean and avoid non-value-added activities when evaluating content quality against those requirements?

  1. Design your requirements for specific personas. It’s not unreasonable to expect a certain level of expertise from anyone in your content supply chain, be it copywriters, technical writers, editors, subject matter experts, translators, reviewers, revisers, or proofreaders. This expertise may include a certain level of language proficiency, general industry domain knowledge, and professional experience (including knowledge of industry standards).
  2. Refer to pre-existing requirement sources (e.g. public style guides). Software engineers using Object-Oriented Programming techniques (OOP) leverage the power of inheritance extensively when writing their code to avoid duplicated effort and ensure the same behavior in different contexts. You can do the same with your content requirements, too, by referring to “master documents” for everything you don’t want to specify explicitly. Typical references include Chicago Manual of Style, European Commission Directorate-General for Translation’s English Style Guide, and Microsoft Style Guides (there’s a version for English content called Microsoft Manual of Style, as well as multiple versions for localized content in many different languages).
  3. Think of the worst outcome that may happen if you omit a requirement. Not all requirements have the same “quality” to them. That is, your audience (readers and users) might be more sensitive to certain aspects of quality when compared to others. So ask yourself: why do we really put this requirement in place, and what do we stand to gain or lose? What might happen to our content performance if our content producers interpret this aspect differently in every piece of content? Do we have specific data that proves that, or is it just a hunch? (or perhaps the loudest voice in the room?)

How do you currently define & manage content requirements in your organization, and how does your approach change across languages, content types, and departments? What key challenges do you face when trying to communicate your requirements for producing high-quality content to your team, peers, vendors, and stakeholders? How do you know which requirements are essential, and which should rather NOT be there? Share your experience in the comments section!