I’m a big fan of unit tests when developing APIs such as Ruuvi data encoding and decoding functions. The tests let us tie the documentation and output of the code together, as the documented normal cases, edge cases and errors are written out as tests.
The downside is that we cannot use the same tools we’re familiar with when integration testing entire program, such as our embedded device IDE debugger when we want to debug the unit tests themselves.
While printing out messages while in program does have it’s place, such as when executing code with hard real-time requirements where having a debugger stop execution leads to a failure of the system, I favor using a proper debugger when possible.
The code example we’re using here are a work-in-progress communication de-/encoders for Ruuvi Gateway nRF52 and ESP32 communication, available at GitHub in ruuviblog-ceedling branch.
Structure of our project
Most of my C-projects nowadays have same structure:
- .github folder where our GitHub Actions reside for running the tests, Doxygen generation, style enforcing etc automatically before code can be merged to master.
- build folder where the outputs of uni test framework Ceedling will be stored
- src folder where the actual source code is.
- test folder for storing the unit test code.
- Various configuration files for the tools and documentation at root level
Running the debugger
Ceedling works internally by building an executable of each test and checking the output of the executable. This gives us excellent an entry point into our test.
Let’s pick the last failed test case, test_ruuvi_endpoint_ca_uart_adv_decode. This is related to decoding serialized Bluetooth advertisement scan report into structured object with MAC address of sender, RSSI and payload data.
void test_ruuvi_endpoint_ca_uart_adv_decode (void)
{
re_status_t err_code = RE_SUCCESS;
uint8_t data[] =
{
RE_CA_UART_STX,
41U + CMD_IN_LEN,
RE_CA_UART_SET_PHY,
0xC9U, 0x44U, 0x54U, 0x29U, 0xE3U, 0x8DU, RE_CA_UART_FIELD_DELIMITER, //!< MAC
0x02U, 0x01U, 0x04U, 0x1BU, 0xFFU, 0x99U, 0x04U, 0x05U, 0x0FU, 0x27U, 0x40U,
0x35U, 0xC4U, 0x54U, 0x00U, 0x50U, 0x00U, 0xC8U, 0xFCU, 0x20U, 0xA4U, 0x56U,
0xF0U, 0x30U, 0xE5U, 0xC9U, 0x44U, 0x54U, 0x29U, 0xE3U, 0x8DU,
RE_CA_UART_FIELD_DELIMITER, //!< Data
0xD8U, RE_CA_UART_FIELD_DELIMITER, //RSSI
RE_CA_UART_ETX
};
re_ca_uart_ble_adv_t expect_params =
{
{ 0xC9U, 0x44U, 0x54U, 0x29U, 0xE3U, 0x8DU }, //!< MAC
{
0x02U, 0x01U, 0x04U, 0x1BU, 0xFFU, 0x99U, 0x04U, 0x05U, 0x0FU, 0x27U, 0x40U,
0x35U, 0xC4U, 0x54U, 0x00U, 0x50U, 0x00U, 0xC8U, 0xFCU, 0x20U, 0xA4U, 0x56U,
0xF0U, 0x30U, 0xE5U, 0xC9U, 0x44U, 0x54U, 0x29U, 0xE3U, 0x8DU
}, //!< Data
31U, //!< Data length
-40 //RSSI
};
re_ca_uart_cmd_t expect_cmd = RE_CA_UART_ADV_RPRT;
re_ca_uart_payload_t payload = {0};
err_code = re_ca_uart_decode (data, &payload);
TEST_ASSERT (RE_SUCCESS == err_code);
TEST_ASSERT (!memcmp (&expect_params, &payload.params.adv, sizeof (expect_params)));
TEST_ASSERT (!memcmp (&expect_cmd, &payload.cmd, sizeof (expect_cmd)));
}Our test case defines data going to decoder and expected output. Data is parsed and parsed outputs compared to our expectations.
static re_status_t re_ca_uart_decode_adv_rprt (const uint8_t * const buffer,
re_ca_uart_payload_t * const payload)
{
re_status_t err_code = RE_SUCCESS;
if (buffer[RE_CA_UART_LEN_INDEX] != RE_CA_UART_CMD_PHY_LEN)
{
err_code |= RE_ERROR_DECODING;
}
else
{
payload->cmd = RE_CA_UART_ADV_RPRT;
memcpy (payload->params.adv.mac,
buffer + RE_CA_UART_PAYLOAD_INDEX, RE_CA_UART_MAC_BYTES);
payload->params.adv.adv_len = buffer[RE_CA_UART_LEN_INDEX]
- RE_CA_UART_MAC_BYTES - RE_CA_UART_RSSI_BYTES;
memcpy (payload->params.adv.adv,
buffer + RE_CA_UART_PAYLOAD_INDEX + RE_CA_UART_MAC_BYTES,
payload->params.adv.adv_len);
payload->params.adv.rssi_db = u8toi8 (
buffer[RE_CA_UART_PAYLOAD_INDEX
+ buffer[RE_CA_UART_LEN_INDEX]
- RE_CA_UART_RSSI_BYTES]);
}
return err_code;
}
re_status_t re_ca_uart_decode (const uint8_t * const buffer,
re_ca_uart_payload_t * const payload)
{
re_status_t err_code = RE_SUCCESS;
// Sanity check buffer format
if (NULL == buffer)
{
err_code |= RE_ERROR_NULL;
}
else if (NULL == payload)
{
err_code |= RE_ERROR_NULL;
}
else if (RE_CA_UART_STX != buffer[RE_CA_UART_STX_INDEX])
{
err_code |= RE_ERROR_DECODING;
}
else if (RE_CA_UART_ETX != buffer[buffer[RE_CA_UART_LEN_INDEX] + RE_CA_UART_HEADER_SIZE])
{
err_code |= RE_ERROR_DECODING;
}
else if (RE_CA_UART_NOT_CODED != payload->cmd)
{
err_code |= RE_ERROR_INVALID_PARAM;
}
else
{
switch (buffer[RE_CA_UART_CMD_INDEX])
{
case RE_CA_UART_SET_FLTR:
err_code |= re_ca_uart_decode_set_fltr (buffer, payload);
break;
case RE_CA_UART_CLR_FLTR:
err_code |= re_ca_uart_decode_clr_fltr (buffer, payload);
break;
case RE_CA_UART_SET_CH:
err_code |= re_ca_uart_decode_set_ch (buffer, payload);
break;
case RE_CA_UART_SET_PHY:
err_code |= re_ca_uart_decode_set_phy (buffer, payload);
break;
case RE_CA_UART_ADV_RPRT:
err_code |= re_ca_uart_decode_adv_rprt (buffer, payload);
break;
default:
err_code |= RE_ERROR_DECODING;
break;
}
}
return err_code;
}Our code does a few sanity checks and then passes data on for correct parser as determined by payload type included in data.
To get a better idea of what went wrong we run
lldb build/test/out/test_ruuvi_endpoint_ca_uart.outwhich opens the test file in lldb. GDB works just as well if that’s your preference.
(lldb)type format add --format hex uint8_t
(lldb) breakpoint set --file ruuvi_endpoint_ca_uart.c --line 150
(lldb) breakpoint set --file test_ruuvi_endpoint_ca_uart.c --line 411
(lldb) runWe set breakpoints in the test file before the line which runs the decoder and in the decoder to the start of sanity checks.
The first bug is now clear, in the line 8 of test function gist above we setup wrong data and the parser correctly returns error code which triggers assert and fails the test.
Fixing the bugs
From now on, fixing the bugs is an iterative process. We fix the test data, run the test again only to run into buggy length validation in the decoder. We fix the length validation, and see that RSSI field gets fixed delimiter character instead of actual RSSI value. We fix the RSSI field to notice that advertisement length is calculated incorrectly leading to buffer overflow and still overwriting RSSI field. We fix the overflow etc etc… Until finally the test passes.
Just because the test passes it doesn’t mean the code is perfect. We add a test case which doesn’t have the delimiters in data and start fixing new buffer overflows. Eventually we’re satisfied with the corner cases tested and are ready to check that the function is covered by our test cases.
ceedling gcov:all utils:gcovWith this command we have created a HTML report of our test execution and we can check that our tests actually cover what we expect them to.
It turns out that removing the last delimiter breaks our assumptions about the positioning of elements inside the data and an earlier check stops the decoding, and our test doesn’t verify bit errors on last delimiter. Let’s fix that too.
Conclusion
Unit testing is a powerful tool in reducing bugs in our code, but to get full benefit of the unit testing we need to have a separate set of tools to verify that our tests actually test what we expect and to debug the tests themselves.
Here we went through how to use LLDB or GDB to debug the bugs in Ceedling. If you’re interested in more code quality related posts let me know in Ruuvi Slack or Forums and I’ll write follow-ups.
These posts are supported by your purchases of RuuviTags, if you enjoy them please go measure your world!
