Benefits

Live data, such as from a production environment, can be a rich source that is large in scale and contains a variety of data characteristics. If tests pass using production data, at a minimum you can expect that code changes won’t break production. That’s pretty good.

When working with data where features change unpredictably or with complex data structures that are difficult to create synthetically, working with live data is likely to give you more accurate test results.

Challenges

Corrupting production data with testing activities is a risk you absolutely must avoid when using live data. A former colleague shared a story in which some well-meaning developers at her company added production credentials to unit-test some data pipeline code locally. As a result of running the unit tests, the developers accidentally deleted all the production data in the most important table, in the product’s most central service, resulting in a total outage and major reputational damage.

For this reason, I recommend limiting live data use to read-only. If tests need to create, modify, or delete data, you’re better off using a fake, such as a test database, or a mock.

When testing with live data, additional steps are needed to connect to and authenticate with data sources. Depending on company policies and the type of data involved, it can take time to get access approval and, potentially, additional time to anonymize data for testing purposes. You’ll also need to configure local development environments to access these resources and make sure test code can authenticate and connect.

As an example, I was running a test suite that connected to an Elastic instance in a test environment. If the Elastic connection was down, the code under test would retry for five minutes. Several times I wondered why my pytest session was hanging, ultimately realizing it was retrying the Elastic connection because I forgot to connect or the tunnel went down while testing. In another case, when working with Medicaid data, getting approval to access data could be an endeavor that took weeks or months. With only live data as a testing option, I wouldn’t have been able to meaningfully contribute during that time.

Data privacy is a significant concern when working with production data. Throughout this book, I’ve shared some examples where I’ve worked with medical data, including the ephemeral test environment approach in Chapter 5, which our team used to test against production data due to the strict privacy rules.

In some cases, data may need to be anonymized for testing, where sensitive information such as PII or PHI is removed or obfuscated. If you find yourself in this situation, stop and consult your company’s legal and security teams.

In some cases, the process of anonymizing data removes essential data characteristics needed for accurate testing. If you need to test with anonymized data, compare it to production to ensure that data characteristics are consistent before betting test results on it.

When testing with live data, another thing to consider are the impacts of testing on the resource serving or storing the data. For example, in Figure 8-2 the HoD social media database is processing and serving content for the HoD website. Performance for the website could be impacted with additional load on the database for testing. In these cases, having dedicated read replicas or snapshots available for testing is a good alternative.

A lower-cost alternative to replicas or full snapshots are partial data dumps, providing only the subset of data used for testing over a shortened time period. You can use the test database technique from Chapter 8 and load the database dump using test fixtures.

I’ve worked with systems in which a job is set up to periodically refresh test data, storing a compressed object in cloud storage that is limited to the tables and fields that are of interest for testing. This approach has a smaller storage footprint than a full database dump and does not require database replicas or managing database connections in your unit-testing code. In addition, the refresh cycle kept data up to date with the current state of production.

Working with live data when testing can also increase your bills. Keep this in mind especially when testing data extraction logic. Data pulled from cloud storage will incur costs for egress and object access. When working with third-party data sources, keep in mind how test loads could impact quotas, limits, and costs. For example, I worked on a system that synced data from a content management system (CMS). The CMS had pretty paltry API quotas, so when making code changes, we had to be careful not to exhaust our daily allotment.

Finally, your software development lifecycle may not support live data use. Consider whether you are developing a pipeline that ingests data from a data source maintained by another team at your company. This is an issue I’ve dealt with in the past, where the data source team was trying to meet a deliverable that changed the data schema before the pipeline team had time to implement the schema changes on the ingestion side. The pipeline team was accustomed to using the live data source, but they had to use synthetic data in the interim while adapting to the schema changes. Building in a plan for synthetic data creation can be a good backup strategy so that you won’t be scrambling if a situation like this occurs.

Benefits

One of the superpowers of synthetic data is that you aren’t limited to the data available in production. That may sound sort of odd: limited by production data? Production data doesn’t always include all the possible cases pipeline code is designed to operate against. With synthetic data, you can ensure that all the important data code paths are tested.

As an example, I worked on a project that got permission from a customer to use their production dataset for testing. While the dataset was large, it didn’t contain the right data characteristics to trigger a performance issue we were debugging in our database queries. This highlights another benefit of synthetic data in that you don’t have to get customer permission.

Another benefit of synthetic data is it can help communicate important data characteristics to other developers. With synthetic data included in your test suite, a new developer can quickly come up to speed without having to gain access to remote systems where live data is stored. You also bypass the kinds of connectivity and authentication issues I mentioned in “Working with Live Data”.

Eliminating connections to data sources saves costs, as you aren’t running queries against a database or accessing cloud storage, and it enables unit tests to be easily added in CI. Later in this chapter, you’ll see how schema-driven data generation and CI testing catch a data pipeline bug before it hits production.

One of my favorite uses of synthetic data is regression testing, where tests are added to ensure that prior functionality hasn’t been lost. When a data-related bug arises in production, you can reproduce the failure in testing by adding test data that emulates the data that caused the failure. As an aside, if you aren’t already in the practice of reproducing bugs in tests before fixing them, I highly recommend it. Not only will you feel like Sherlock Holmes for isolating the root cause and reproducing the failure, you will be incrementally making your testing more robust.

Finally, the more you can test with synthetic data and unit tests, the less cost and complexity will be associated with testing. Depending on live data or snapshots of live data requires additional cost and overhead for accessing these systems and maintaining snapshots. Even if you can cheaply create a snapshot of data to load into a local database for testing, consider the additional work involved in keeping that development database up to date. In my experience, this often leads to inadvertently testing against stale data and the “it works on my machine” conundrum.

Challenges

While you may find it straightforward to create synthetic data for small-size, low-complexity data sources, this process can become difficult or impossible as data gets larger and more complicated.

It is very easy to go down a rabbit hole¹ when creating synthetic data. Fundamentally, you are trying to come up with a model of the source data. When getting started, you’ll naturally gravitate to modeling the most straightforward cases, but things can get complicated as corner cases are encountered that don’t fit the model.

I don’t know about you, but for me, when I’ve gotten most of the way through a problem and uncovered some thorny corner cases, I can easily become enamored with finding a solution. As a matter of fact, this happened to me while trying to come up with examples for this chapter. As a result, I created a model that I wouldn’t use for development but that was very intellectually satisfying to figure out.

I encourage you to resist this temptation. If you find yourself creating a lot of one-off special cases or spending more than a day trying to model a data source, stop and reevaluate whether synthetic data is the right choice. The next section will provide some guidance for this.

Another challenge can be overdoing it on the data model and creating cases that either aren’t relevant or overexercise the code under test. When creating synthetic data, do this side by side with the logic the data will exercise. This will help you home in on the data cases that are relevant to the code under test, keeping you from going overboard with test data generation.

There’s an excellent chance synthetic data will quickly become obsolete and unusable if it requires constant maintenance to stay up to date. Keep in mind the maintenance required when developing synthetic data. You’ll see some techniques in this chapter to automate the data generation process, which will help reduce maintenance needs.

Is Synthetic Data the Right Approach?

Synthetic data can be a real boon to your testing process, but it can be disastrous if it is applied in the wrong situations. In my experience, synthetic data is most valuable when one or more of the following is true:

• It models well-characterized, stable data sources.

• It is kept up to date automatically or with a minimum of effort.

• It is used to test well-bounded logic.

To get a sense of whether synthetic data is right for a given data dependency, consider these questions:

What are the consequences of inaccuracies in the synthetic data?

Consider the role synthetic data plays in pipeline fidelity and data quality. The more critical testing is in this process, the greater the risk of issues arising from mistakes in synthetic data.

For example, when working with an API data source for a pipeline project, I used both synthetic data and JSON schemas together to ensure that the data was transformed correctly. I used the synthetic data in unit tests to verify handling of different API responses, while the JSON schema was used during runtime to validate that the API response was as expected.

At some point, the API response changed. The unit tests did not catch this change because the synthetic data was not up to date, but the pipeline validation did catch it. Because I was using both unit tests and data validation, as you saw in Chapter 6, the out-of-date synthetic data did not result in a data-quality issue in the pipeline. This is also a good example of how data validation and unit testing complement each other in ensuring pipeline fidelity.

If instead you have a situation where tests are the primary mechanism for ensuring data quality, be absolutely sure that any synthetic data used is accurate. It can be a better idea to use real data in this case.

How complex is the data source you want to model? How stable is the data format?

Continuing with the API example, this was a good candidate for synthetic data for the following reasons:

• There were only a handful of potential responses.

• The data structure was consistent across responses.

• The API was developed internally.

With a limited number of distinct values to model and a consistent data structure, the API response was low effort to create and maintain. An additional benefit was that the API was developed by another team at the company, so it was possible to share API response schema information. By linking the schema with the synthetic data generation, I could guarantee that the test data was an accurate model.

What is involved in maintaining the synthetic data? Is the benefit worth the time investment?

If synthetic data is high maintenance, consider how likely it is that it will be kept up over time. Based on my experience, I suspect there is a vast graveyard of well-intentioned synthetic data that fell by the wayside due to lack of maintenance.

For the rest of the chapter, you’ll see different techniques for creating synthetic data to test parts of the HoD survey pipeline from Chapter 7, repeated here in Figure 9-1.

Figure 9-1. HoD batch survey pipeline

The synthetic data approaches presented are based on building some simple models of live data samples. There are more advanced techniques that I won’t cover, such as creating data through statistical processes, artificial intelligence, or other algorithmic means. There are entire products and services that focus on creating synthetic data, as well as books and journal articles on the topic.

The data you’ll see modeled in the rest of the chapter is the source data for the survey pipeline, the Field survey data stored in the Survey bucket in Figure 9-1. An example of the survey data is shown in Table 9-1 to provide a sense of the data characteristics.

The survey dataset contains a row for each sighting recorded by a user of the bird survey app. This includes the user’s email and location and a freeform description. Users can attach pictures to each sighting, which are stored by the survey app in a cloud storage bucket, the links to which are listed in the “Image files” column. Users can also provide an approximate count of the number of birds sighted.

The survey data is transformed in the “Extract species” step in Figure 9-1. The transformation looks for specific heron species within the Description field and creates a new column, Species, that contains the matching species, or null if no match is found. Table 9-2 shows the results of the survey data in Table 9-1 after it is transformed by the “Extract species” step.

This transformation is performed by the apply_species_label method, as depicted in the following code, which you can find in transform.py. You’ll run this code later as part of the unit test; for now, just get a sense of how it works:

species_list = ["night heron", "great blue heron", "gray heron", 
                "whistling heron"]

def apply_species_label(species_list, df):
   species_regex = f".*('|'.join(species_list)}).*"
   return (df
       .withColumn("description_lower", f.lower('description'))
       .withColumn("species", f.regexp_extract('description_lower',
                                                species_regex, 1))
       .drop("description_lower")
   )

To extract the species label, a list of species, species_list, is combined into a regular expression (regex). This regex is applied to a lowercase version of the description column, description_lower. The first substring to match an element in species_list is extracted as the species column. Once the species has been determined, the description_lower column is dropped as it is not needed anymore.

Synthetic Data Libraries

One approach to creating synthetic data is to use a library. Faker is a Python library that generates synthetic data, from primitive types such as boolean and integer to more sophisticated data such as sentences, color, and phone numbers, to name a few. If you don’t find what you need among the standard and community providers, try creating your own. A provider is the Faker term for a fake data generator.

The method basic_fake_data in faker_example.py shows how to use Faker to create synthetic data to test apply_species_label. This example uses several Faker built-in providers, including email, latitude and longitude, filepath, and words:

def basic_fake_data():
   fake = Faker()
   fake_data = {
       "user": fake.email(),
       "location": fake.local_latlng(),
       "img_files": [f"s3://bucket-name{fake.file_path(depth=2)}"],
       "description": f"{' '.join(fake.words(nb=10))}",
       "count": random.randint(0, 20),
   }
   for k,v in fake_data.items():
       print(k, ":", v)

Try running this code by opening a Python terminal in the virtual environment in the testing directory. Keep in mind that the results are random; the results printed here will be different from what you see when running the fake data generators:

(my_venv)$ cd testing
(my_venv)$ python
. . .
>>> from faker_example import basic_fake_data
>>> basic_fake_data()
user : lindseyanderson@example.org
location : ('41.47892', '-87.45476', 'Schererville', 'US', 'America/Chicago')
img_files : ['s3://bucket-name/quality/today/activity.txt']
description : another voice represent work authority daughter best dream name 
              meeting
count : 13

Looking at basic_fake_data, notice that there isn’t a list of different test cases, as there is with the manual data generation approach. Instead, fake_data is a dictionary that can be created any number of times to generate new test cases. In the terminal, call basic_fake_data again and you’ll see that a new set of synthetic data is created:

>>> basic_fake_data()
user : wilsonbront@example.net
location : ('39.78504', '-85.76942', 'Greenfield', 'US', 
            'America/Indiana/Indianapolis')
img_files : ['s3://bucket-name/sure/ground/establish.pptx']
description : budget panda animal act visit agent important half respond stuff
count : 8

Another benefit of this automated approach is that you don’t need to manually update all the test cases if there is a change to the data schema. Instead, you just update the Faker definitions. For example, let’s say the user field changes from an email address to a UUID. In the basic_fake_data function, change fake.email() to fake.uuid4(). To see this change in the terminal, you’ll need to quit and restart Python and then reimport the module. Now when you run basic_fake_data, you can see the UUID instead of the email for the user field:

(my_venv)$ python
>>> from faker_example import basic_fake_data
>>> basic_fake_data()
user : b7e61421-b7e5-4266-bc3f-2e3575358d32
location : ('41.47892', '-87.45476', 'Schererville', 'US', 'America/Chicago')
img_files : ['s3://bucket-name/quality/today/activity.txt']
description : another voice represent work authority daughter best dream name 
              meeting
count : 13

def description(self):
    species = fake.words(ext_word_list=util.species_list, nb=randint(0,1))
    word_list = fake.words(nb=10)
    index = randint(0, len(word_list)-1)
    description = ' '.join(word_list[0:index] + species + word_list[index+1:])

>>> from faker_example import DescriptionProvider
>>> from faker import Faker
>>> fake = Faker()
>>> fake.add_provider(DescriptionProvider)
>>> fake.description()
'leave your stop fast sport can company turn degree'
>>> fake.description()
'card base environmental very computer seek view skin assume'
>>> fake.description()
'choose sense night student night heron two it item mouth clearly'

def create_test_data_with_manual(length):
   data, expected = manual_example.create_test_data()
   auto_data, auto_expected = create_test_data(length)
   return (data+auto_data, expected+auto_expected)

def description_distribution(self):
       . . .
       cases = ['', species, species_part, words_only, with_species, with_part]
       weights = [10, 2, 3, 5, 6, 7]
       return choices(cases, weights=weights)

from string import digits
def generate_id():
   id_values = random.sample(digits, 5)
   return f"bird-surv-{''.join(id_values)}"

def create_test_data_ids(length=10):
   mock_data = []
   for _ in range(length):
       mock_data.append(
           {
               "id": generate_id(),
               "user": fake.email(),
        . . .

>> import util
>> from faker_example import create_test_data_ids
>> create_test_data_ids()
[{'id': 'bird-surv-26378', 'user': '[email protected]', 'location': ...

Schema-Driven Generation

So far, you’ve seen how to create a synthetic data generator that can mimic the size and distribution of values from a source dataset. The data generation happens at test time, so there is no data to maintain or store. While this provides some benefits over manual data, there’s an outstanding question of what happens when the source dataset changes. How will you know the synthetic data generator is out of date?

As described in Chapter 4, schemas are a powerful tool for data validation. Schemas are also useful when creating test data; linking the data generation process to schemas will help you track changes in source data. In this section, you’ll see how to link the automated data generation techniques in “Synthetic Data Libraries” to a schema to further automate test data generation.

Ideally you have access to explicit data schemas, meaning definitions that are used as part of the data pipeline to describe the data being processed. These could be represented as JSON or Spark schemas. If explicit schemas aren’t available, you can extract schemas from a data sample for testing. mock_from_schema.py provides an example of how to do this for a Spark DataFrame:

def create_schema_and_mock_spark(filename, length=1):
   with open(filename) as f:
       sample_data = json.load(f)
   df = spark.sparkContextparallelize(sample_data).toDF()  
   return generate_data(df.schema, length)

With the DataFrame loaded, df.schema gives you the schema, which can be used for the data generation technique in this section:

StructType(List(
   StructField(count,LongType,true)
   StructField(description,StringType,true)
   StructField(img_files,ArrayType(StringType,true),true)
   StructField(location,ArrayType(StringType,true),true)
   StructField(user,StringType,true)))

A JSON version of the schema can be obtained by calling df.schema.json().

When working with Pandas, you can extract a JSON schema using build_table_schema, but keep in mind that the schema inference provided by Spark gives a richer view of the underlying data types. Consider this Pandas schema generated from the same sample data as that used for the preceding Spark schema:

'fields': [{'name': 'index', 'type': 'integer'},
  {'name': 'user', 'type': 'string'},
  {'name': 'location', 'type': 'string'},
  {'name': 'img_files', 'type': 'string'},
  {'name': 'description', 'type': 'string'},
  {'name': 'count', 'type': 'integer'}],

Notice that img_files and location are listed as string types, while their contents are an array of strings. There is also no nullable information.

DYNAMIC_GENERAL_FIELDS = {
   StringType(): lambda: fake.word(),
   LongType(): lambda: fake.random.randint(),
   ArrayType(StringType()): lambda: [fake.word() 
                                     for i in range(fake.random.randint(0,5))]
}

DYNAMIC_NAMED_FIELDS = {
   ("count", LongType()): lambda: fake.random.randint(0, 20),
   ("description", StringType()): lambda: fake.description_only(),
   ("user",StringType()): lambda: fake.email(),
}

def get_value(name, datatype):
   if name in [t[0] for t in DYNAMIC_NAMED_FIELDS]:
       value = DYNAMIC_NAMED_FIELDS[(name, datatype)]()
       return value

   if datatype in DYNAMIC_GENERAL_FIELDS:
       return DYNAMIC_GENERAL_FIELDS.get(datatype)()
      
   raise DataGenerationError(f"No match for {name}, {datatype}")

def generate_data(schema, length=1):
   gen_samples = []
   for _ in range(length):
gen_samples.append(tuple(map(lambda field: get_value(field.name, field.dataType), 
                             schema.fields)))

   return spark.createDataFrame(gen_samples, schema)

$ cd testing
$ python
>>> from schemas import survey_data
>>> from mock_from_schema import generate_data
>>> df = generate_data(survey_data, 10)
>>> df.show(10, False)

(my_venv)$ cd testing
(my_venv)$ pytest -v -k test_transform_schema -s

T.StructField("description",T.ArrayType(T.StringType(), True), True)

mock_from_schema.DataGenerationError: No field mapping found for description, 
      ArrayType(StringType,true)