SCRAP: Serial Compact Reliable Asynchronous Protocol

Version 0.8.1

Table of Contents

• SCRAP: Serial Compact Reliable Asynchronous Protocol
  • Table of Contents
  • Introduction
  • Features
  • Protocol Overview
    • Packet Structure
    • Framing (COBS)
    • Reliability (Stop-and-Wait)
  • Getting Started (C Implementation)
    • 1. File Structure
    • 2. Configuration (scrap_config.h)
    • 3. Integration into your Application
  • C API Reference
    • Core Functions
    • Sending Data
    • Buffer Sizing Macros
  • Python Implementation
    • Installation
    • Python Usage Example
  • Building and Running the Examples
  • License

Introduction

SCRAP is a lightweight, packet-based serial communication protocol designed specifically for resource-constrained embedded systems like microcontrollers. It provides a reliable, easy-to-use mechanism for exchanging structured data over mediums like UART, RS-232, or virtual serial ports.

This library is offered in two fully interoperable implementations:

1. C (C89): Written in pure, dependency-free ANSI C for maximum portability. It features a highly memory-efficient architecture, making it ideal for bare-metal MCUs with limited RAM.
2. Python 3: A clean, easy-to-use implementation for PC-side applications, scripts, and testing tools.

Features

• Reliable Delivery: Implements a Stop-and-Wait ARQ (Automatic Repeat reQuest) mechanism with timeouts and retries to guarantee packet delivery.
• Extremely Memory Efficient (C): The C library is designed to operate with minimal static RAM, using just two primary user-supplied buffers. No dynamic memory allocation (malloc) is ever used.
• Portable: The C library is written in strict C89 (ANSI C) and has no external dependencies. The Python library depends only on pyserial and cobs.
• Robust Framing: Uses Consistent Overhead Byte Stuffing (COBS) to unambiguously frame packets, preventing synchronization errors.
• Data Integrity: A CRC16-CCITT checksum is appended to every packet to detect and discard corrupted data.
• Flexible Payload: Supports both generic DATA frames and structured COMMAND frames, allowing for both simple data streaming and RPC-style communication.
• User-Controlled: The user provides all key components: memory buffers, a physical-layer write function, and a timestamp function, giving complete control over the system integration.

Protocol Overview

Packet Structure

Every SCRAP packet, before encoding, follows a standard structure:

Header (6 bytes)	Payload (payload_len bytes)	CRC (2 bytes)
seq (2) | flags (1) | type (1) | payload_len (2)	Application Data or Command	CRC16-CCITT

• Sequence Number: A unique ID for each packet.
• Flags: A bitfield indicating if the packet requires an ACK (FLAG_ACK_REQUIRED) or if it is an ACK (FLAG_IS_ACK_PACKET).
• Frame Type: SCRAP_FRAME_TYPE_DATA or SCRAP_FRAME_TYPE_COMMAND.
• Payload Length: The size of the payload field in bytes.
• Payload: The actual application data. For commands, this is further structured with a 2-byte Command ID followed by parameters.
• CRC: A 16-bit checksum of the header and payload.

Framing (COBS)

To handle the "framing problem" (knowing where a packet begins and ends), SCRAP uses COBS. The entire raw packet (Header + Payload + CRC) is COBS-encoded. This process removes all 0x00 bytes from the data.

A special 0x00 byte (SCRAP_PACKET_DELIMITER) is then appended to the encoded data to unambiguously mark the end of the frame. This makes the receiver's job simple and robust.

Reliability (Stop-and-Wait)

When a packet is sent "reliably" (reliable = SCRAP_TRUE), the sender starts a timer and waits for an ACK packet from the receiver.

• If the ACK is received in time, the transmission is successful.
• If the ACK is not received before the timeout, the sender retransmits the original packet.
• This is repeated up to max_retries times before declaring the transmission a failure.

This simple mechanism guarantees that a packet is received, at the cost of only allowing one "in-flight" reliable packet at a time.

Getting Started (C Implementation)

Integrating the C library into your embedded project is a three-step process.

1. File Structure

Copy the following files into your project's source directory:

• scrap.h (The public API)
• scrap.c (The implementation)
• scrap_config.h (Your project-specific configuration)

2. Configuration (scrap_config.h)

This is the central configuration file. You must define your system's native types and can optionally enable logging or provide custom memory functions.

/**
 * @file scrap_config.h
 * @brief User configuration file for the SCRAP library.
 */
#ifndef SCRAP_CONFIG_H_
#define SCRAP_CONFIG_H_

#include <stdint.h>  /* For standard integer types */
#include <stdbool.h> /* For standard boolean types */

/* --- 1. Debug Logging (OPTIONAL) --- */
/* To enable debug logging, uncomment this line. */
/* #define SCRAP_LOG(fmt, ...) printf("[SCRAP-DEBUG] " fmt "\n", ##__VA_ARGS__) */


/* --- 2. Portability & Type Definitions (MANDATORY) --- */
#define SCRAP_U8    uint8_t
#define SCRAP_U16   uint16_t
#define SCRAP_TRUE  true
#define SCRAP_FALSE false

/* --- 3. Memory Function Overrides (ADVANCED & OPTIONAL) --- */
/* The library defaults to using standard memcpy, etc. from <string.h> */
#ifndef SCRAP_MEMCPY
    #define SCRAP_MEMCPY(dst, src, len)  memcpy(dst, src, len)
#endif
// ... and so on for memset and memmove

#endif /* SCRAP_CONFIG_H_ */

3. Integration into your Application

Here is a minimal example of how to initialize and use the library in your main.c.

#include "scrap.h"
#include "my_uart_driver.h" // Your hardware driver
#include "my_systick_driver.h" // Your timer driver

// Define the maximum payload your application will ever use.
#define MY_APP_MAX_PAYLOAD 128

// 1.a. Use the helper macro to declare correctly-sized buffers.
SCRAP_DEFINE_BUFFERS(my_scrap_instance, MY_APP_MAX_PAYLOAD);

// 1.b. Declare memory for the handle
static uint8_t scrap_handle_memory[SCRAP_HANDLE_STATIC_SIZE];

// 2. Implement the transport and timestamp functions.
uint16_t my_transport_write(const uint8_t* data, uint16_t len) {
    return uart_write_blocking(data, len);
}

uint16_t my_get_timestamp_ms(void) {
    return systick_get_milliseconds();
}

// 3. Implement application callbacks (optional but recommended).
void my_data_received_callback(Scrap_Handle* h, const uint8_t* payload, uint16_t len) {
    printf("Received %u bytes of data!\n", len);
}

void my_send_success_callback(Scrap_Handle* h, uint16_t seq_num) {
    printf("Packet #%u was successfully delivered!\n", seq_num);
}

int main(void) {
    // Hardware initialization
    uart_init();
    systick_init();

    // 4. Configure the library.
    Scrap_Config config = {
        .rx_buffer = my_scrap_instance_rx_buffer,
        .rx_buffer_size = sizeof(my_scrap_instance_rx_buffer),
        .tx_buffer = my_scrap_instance_tx_buffer,
        .tx_buffer_size = sizeof(my_scrap_instance_tx_buffer),
        .transport = {
            .transport_write = my_transport_write,
            .get_timestamp_ms = my_get_timestamp_ms
        },
        .callbacks = {
            .data_received = my_data_received_callback,
            .send_success = my_send_success_callback
        }
    };

    // 5. Initialize the handle.
    Scrap_Handle* scrap_handle = Scrap_Init(scrap_handle_memory, &config);
    if (!scrap_handle) {
        // Handle initialization error
        return -1;
    }

    // 6. Send a reliable data packet.
    const char* message = "Hello from SCRAP!";
    Scrap_SendData(scrap_handle, (const uint8_t*)message, strlen(message), SCRAP_TRUE);

    // 7. In your main loop, continuously feed received bytes and process the handle.
    while (1) {
        if (uart_has_data()) {
            uint8_t byte = uart_read_byte();
            Scrap_ReceiveBytes(scrap_handle, &byte, 1);
        }
        Scrap_Process(scrap_handle); // Handles timeouts and retransmissions
    }
}

C API Reference

Core Functions

• size_t Scrap_GetHandleSize(void): Returns the required size for the handle memory.
• Scrap_Handle* Scrap_Init(void* h_mem, const Scrap_Config* config): Initializes the library.
• void Scrap_ReceiveBytes(Scrap_Handle* h, const SCRAP_U8* data, SCRAP_U16 len): Feeds raw received bytes to the parser.
• void Scrap_Process(Scrap_Handle* h): Must be called periodically to handle time-based events.
• SCRAP_U8 Scrap_IsReadyToSend(Scrap_Handle* h): Checks if a new reliable transmission can be started.

Sending Data

• SCRAP_U16 Scrap_SendData(h, payload, len, reliable): Sends a generic data packet.
• SCRAP_U16 Scrap_SendCommand(h, cmd_id, params, params_len, reliable): Sends a structured command packet.

Buffer Sizing Macros

• SCRAP_CALC_MAX_RAW_PACKET_SIZE(max_payload)
• SCRAP_CALC_MAX_ENCODED_SIZE(raw_size)
• SCRAP_CALC_BUFFER_SIZE(max_payload)
• SCRAP_DEFINE_BUFFERS(instance_name, max_payload_size)

Python Implementation

The Python implementation mirrors the C library's logic and is fully compatible. It's ideal for writing PC-side applications that communicate with an embedded device running the C library.

Installation

The Python library can be installed directly from the source directory using pip.

# Navigate to the root of the repository
# Install the library in editable mode (-e)
pip install -e .

This command uses the setup.py file to install the scrap-protocol package and its dependencies (pyserial and cobs). Using -e is recommended for development, as changes to the source code will be immediately reflected.

Python Usage Example

This example shows a simple PC-side application that sends a "ping" command and waits for a data response from an embedded device.

import serial
import time
from scrap.protocol import Scrap # Import the main class

SERIAL_PORT = 'COM3' # Or '/dev/ttyUSB0' on Linux
BAUDRATE = 115200

# --- Application-specific constants ---
CMD_PING_DEVICE = 0x1001
CMD_PONG_RESPONSE = 0x1002

class PingClient:
    def __init__(self):
        self.device_responded = False
        try:
            self.ser = serial.Serial(SERIAL_PORT, BAUDRATE, timeout=0.01)
        except serial.SerialException as e:
            print(f"Error opening serial port: {e}")
            exit()

        # 1. Instantiate the Scrap protocol handler
        self.scrap = Scrap(transport_write=self.ser.write)

        # 2. Register callbacks
        self.scrap.on_command_received = self.handle_command
        self.scrap.on_send_success = lambda seq: print(f"Ping #{seq} delivered successfully!")
        self.scrap.on_send_failed = lambda seq: print(f"Ping #{seq} failed to deliver!")

    def handle_command(self, command_id, params):
        if command_id == CMD_PONG_RESPONSE:
            print(f"Pong received! Device says: {params.decode('utf-8')}")
            self.device_responded = True

    def run(self):
        print("Sending a reliable PING command to the device...")
        self.scrap.send_command(CMD_PING_DEVICE, reliable=True)

        start_time = time.time()
        while not self.device_responded and (time.time() - start_time) < 5:
            # Feed received bytes to the library
            if self.ser.in_waiting > 0:
                data = self.ser.read(self.ser.in_waiting)
                self.scrap.receive_bytes(data)

            # Process timeouts
            self.scrap.process()
            time.sleep(0.01)

        if not self.device_responded:
            print("Test failed: Device did not respond in time.")
        
        self.ser.close()

if __name__ == "__main__":
    client = PingClient()
    client.run()

Building and Running the Examples

The repository includes examples for both C and Python to demonstrate a file transfer.

1. Create a virtual serial port tunnel for testing on a single machine:

   # This requires 'socat' to be installed
   socat -d -d pty,raw,echo=0,link=scrap_h2d pty,raw,echo=0,link=scrap_d2h

2. Compile the C examples:

   make

3. Run the test:
   • In one terminal, start the C receiver: ./device_downloader
   • In another terminal, start the Python sender: python3 -m examples.file_transfer.host_uploader

You can also test C-to-C (./host_uploader) or Python-to-Python communication.

License

SCRAP is released under the MIT License. See the LICENSE file for details.