Applied Test Case Design: Zammad example

I reviewed various open-source Rails apps on my blog and looked their source code. As I already have them on my computer, it would be a good idea to do some test designs on real code examples.

One of the open-source apps that I was reviewed on my blog was Zammad.

Looking around in the source code, I found this method: 

def check_name
  self.firstname = sanitize_name(firstname)
  self.lastname  = sanitize_name(lastname)

  return if firstname.present? && lastname.present?

  if (firstname.blank? && lastname.present?) || (firstname.present? && lastname.blank?)
    used_name = firstname.presence || lastname
    (local_firstname, local_lastname) = User.name_guess(used_name, email)
  elsif firstname.blank? && lastname.blank? && email.present?
    (local_firstname, local_lastname) = User.name_guess('', email)
  end

  check_name_apply(:firstname, local_firstname)
  check_name_apply(:lastname, local_lastname)
end

That has the following test:

describe '#check_name' do
  it 'guesses user first/last name with non-ASCII characters' do
    user = create(:user, firstname: 'perkūnas ąžuolas', lastname: '')

    expect(user).to have_attributes(firstname: 'Perkūnas', lastname: 'Ąžuolas')
  end
end

It could be a perfect start to improve the testing there.

Please note

What I write here should just be read as an exercise about doing test design, but it is not a critique of the testing written in the project nor an assessment of the quality of Zammad.

I just want to apply some systematic test case design to actual code. I know you are a developer so maybe the first reaction would be to refactor this code because maybe you have a different opinion about how it should work. But we are focusing here on learning testing and not programming so we leave the code alone as it would be frozen. Cannot be touched by refactoring, only the test files can be changed.

The third point concerns the public vs. private method. Notice that this code is used in a before_validation, which is a private method. Thus, we cannot test it directly, but we can test it by creating or validating a record.

Keep in mind whenever you read a piece of code or test written in a production-deployed project that, we don’t know the full context of what happened or any constraints and decisions that were made.

Test Conditions

So the first step in a systematic approach would be to identify test conditions.

What is a test condition?

"A test condition is a testable aspect of a test item, such as a function, transaction, feature, quality attribute or structural element identified as a basis for testing"

IEEE International Standard for Software and Systems Engineering

Step 1: Inputs and Outputs

I don’t have the requirement so I will base the entire test design session on the code.

The first step is to ask what are the inputs and outputs of this method that we can change and check?

Here the inputs are:

And the outputs are:

So, while the method check_name is named as a verification, it will also change the firstname and lastname attributes as it is not just a simple boolean method.

But looking more into this method, we notice the following two lines inside: 

elsif firstname.blank? && lastname.blank? && email.present?
      (local_firstname, local_lastname) = User.name_guess('', email)
    end

And there seems like email will influence the outputs of this method. Thus email is also an input.

The final list of inputs is: 

firstname
lastname
email

If you are writing the code, then inputs and outputs are obvious. Still, from the perspective of an API/code design, it is good to always think about what the inputs are and what the output is (side effects are also part of the output and exceptions).

Step 2: Test Conditions

Having a clear idea about the inputs and outputs of the logic that we are trying to test, we can now identify some test conditions that we can test:

Step 3: Deciding what we are going to test

Analyzing the source code, we see that there are specific methods for:

We should rely on these methods being tested independently, so we want to test how they are integrated and used in check_name without focusing on details about them.

(Just in case you are wondering: How can they be checked? It should not be checked by forcing a send to a private method but again via the before-validation filters).

Imagine that they might be stubbed in testing, so you might only assess how their output is being used but not be able to verify their logic when you are writing tests for check_name

To give an example:

Having said all this, what do we want to focus on?

✅ 1. Sanitization of firstname => We want to make sure the sanitization is executed, but we will not going to focus on details and edge cases of sanitization

✅ 2. Sanitization of lastname => The same point goes for this one.

✅ 3. Having both of them present => this is a case that is part of the logic that we want to test

✅ 4. Having both firstname and lastname being nil => also this one is part of the logic that we want to test

✅ 5. Having one of them nil and the other present => is also part of the logic that we want to test

✅ 6. Capitalize first name and last name => Here we only focus on making sure that capitalize is called but will not focus on edge cases

❌ 7. Exceptions raised by invalid or corrupted data => This would be a choice if the risk is high (like this is a very important piece of logic for our users/business/app)

Test Case Design

For each test condition, we will try to identify the most important tests to write.

Sanitization of firstname

We want to provide valid values (no sanitization needed) and invalid values (sanitization required).

When considering how to sanitize the first name, it's important to identify at least one scenario where the input will clearly require sanitization. For example, if a URL is submitted as the first name, it should definitely be flagged for sanitization, as this would be an invalid input.

We want to send a valid case, such as using “Jane” as the first name.

TC1: 
firstname = “https://example.com” 

TC2: 
firstname = “jane”

Sanitization of lastname

The same goes for sanitization of lastname.

TC3: 
lastname = “https://example.com”

TC4: 
lastname = “Doe”

Having both first name and last name present

For this condition, we need to send values in such a way as to make them true and false. 

TC5: 
firstname = “”
lastname = “”

TC6: 
firstname = “John”
lastname = “Doe”

Having both firstname and lastname nil

Here, the test case will match the condition, and we have many non-nil values in the other test cases, which will eventually reduce.

TC7: 
firstname = nil
lastname = nil

Having one of them nil and the other present

Actually this test condition is related to the following two decisions:

To identify the test cases here, there are two techniques:

  1. We can focus on the decision value => thus, we want to make the decision true and false
  2. We can focus on each condition => we want to create the truth table where each condition is made true and false and check all possible combinations

This seems to be an area with low or maybe medium risk and I will apply the first technique, which will give me a reduced number of test cases. 

TC8: making `(firstname.blank? && lastname.present?) || (firstname.present? && lastname.blank?)` false 
firstname = ‘’
lastname = ‘’

TC9: making `(firstname.blank? && lastname.present?) || (firstname.present? && lastname.blank?)` true
firstname = ‘’
lastname = ‘Smith’
email = ‘jane.snow@example.com’

TC10: making `firstname.blank? && lastname.blank? && email.present?` false (but take into consideration this one is on the false branch of the first one)
firstname = ‘’
lastname = ‘’
email = ‘’ 

TC11: making `firstname.blank? && lastname.blank? && email.present?` false
firstname = ‘’
lastname = ‘’
email = ‘jane.snow@example.com’

Capitalize first name and last name

After the name is guessed, we see in the code that it is capitalized.

So in this case we will have two test cases: one valid where the names does not need to be capitalized and one invalid where the names will be capitalized but we need to remain on the branch about both of them not being present on the same time. 

TC12: 
firstname = ‘’
lastname = ‘’ 
email = ‘jane.snow@example.com’

TC13:
firstname = ‘’
lastname = ‘’ 
email = ‘Jane.Snow@example.com’

Refine the test cases and try to reduce them

Looking at all the test cases, some are similar and can be reduced.

I will start with the first test case, adapt it and create some refined test cases:

Refined Test Case 1

Inputs: 
    firstname = “jane” 
    lastname = “smith”
    email = "something.different@example.com"
Output
    validation success
    firstname = 'jane'
    lastname = 'smith'

This one will check the previous: TC2, TC4, TC6

Refined Test Case 2

Inputs: 
    firstname = “https://example.com” 
    lastname = “https://example.com”
    email = "jane.snow@example.com"
Output
    validation success
    firstname = 'Jane'
    lastname = 'Snow'

This one will check the previous TC1, TC3, TC8, TC11, TC12

Refined Test Case 3

Inputs: 
    firstname = “”
    lastname = “”
    email = "Jane.Snow@example.com"
Output
    validation success
    firstname = 'Jane'
    lastname = 'Snow'

This one will check the previous TC5, TC13

Refined Test Case 4

Inputs: 
    firstname = nil
    lastname = nil
    email = "nil"
Output
    validation failed
    firstname = nil
    lastname = nil

This one will check the TC7, TC10

Refine Test Case 5

Inputs: 
    firstname = ""
    lastname = "Doe"
    email = "jane.snow@example.com"
Output
    validation sucess
    firstname = "Jane"
    lastname = "Snow"

This one will check TC9

Conclusion

In the end, with 5 test cases, we managed to cover many use cases and test conditions. We also made some choices along the way, such as choosing medium-risk coverage instead of high-risk one. This is also part of the idea of good enough testing: to know when you can reduce the test cases and when you should not but try to add more.

One last note:

When examining test case 5 and the code for the check_name and name_guess methods, I am unclear about what the expected last name should be. Should it be "Doe," as the user indicated, or should it match the last name from the email? This uncertainty needs to be clarified with the product team, and it should be incorporated as a test for the name_guess method, rather than being specific to this particular case. Additionally, the behavior of the name_guess method might impact our current method, so once we confirm the expectations with the product team, we can mock that method. This will enable us to ensure it returns the expected value, effectively decoupling our current test from the actual implementation of name_guess, as long as name_guess is a publicly testable method.

PS: All code presented here is taken from open source projects, and I have no affiliation with those particular projects.

References

[IEEE International Standard for Software and Systems Engineering] Software Testing–Part 4: Test Techniques, ISO/IEC/IEEE https://www.iso.org/standard/79430.html