Example of a Complex Integration Module

structs np_udp_functions and np_tcp_functions

The integration interfaces for UDP and TCP communication is probably the most complicated integration. The semantics of each function is not hard to understand but since most operating systems offers blocking system calls they all require an extra wrapper layer to be implemented. Blocking functions are not friendly for embedded system/platforms that needs to be responsive, so Nabto relies on async implementations instead.

Let’s look at an example from the np_tcp.h interface. This interface defines an async_read function to be implemented. The contract for this function is “simple”: Wait on the socket until the sender (other end) sends data to you and once this happens, resolve the completion event. Also, that you need to return the execution thread immediately (it is async) so that the platform can carry on other execution for maximum responsivenes. Here’s the definition:

    /**
     * Read data from a socket.
     *
     * The completion event shall be resolved when a result of the
     * operation is available.
     *
     * @param sock  The socket resource.
     * @param buffer  The buffer to write data to.
     * @param bufferLength  The length of the buffer.
     * @param readLength  The length of received data.
     * @param completionEvent  The completion event to resolve when data has been read.
     */
    void (*async_read)(struct np_tcp_socket* sock, void* buffer, size_t bufferLength, size_t* readLength, struct np_completion_event* completionEvent);

To accomplish this on a posix like blocking system is a little more complicated (when first encountered) than the description/contract says. One way to implement this is to use the select function by which a thread can inform the operating system that it wishes to know if something changes (for example new data is available for read) on a set of file descriptors (sockets). The way to do this is to create a list of both UDP and TCP sockets that needs attention once new data is available on the sockets.

In this implementation example both UDP and TCP sockets are put into a large list of file descriptors and examined via one thread using the operating system select function. Instead a thread for each UDP and TCP could instead have been implemented (which would mean one more allocated thread).

So the overall data structure and flow looks something like this:

Lets look at the implementation of the create function in the tcp module.

np_error_code create(struct np_tcp* obj, struct np_tcp_socket** sock)
{
    struct nm_select_unix* selectCtx = obj->data;
    struct np_tcp_socket* s = calloc(1,sizeof(struct np_tcp_socket));
    s->fd = -1;
    *sock = s;
    s->selectCtx = selectCtx;

    nn_llist_append(&selectCtx->tcpSockets, &s->tcpSocketsNode, s);

    s->aborted = false;
    return NABTO_EC_OK;
}

Only two things happens.

Basic initialization of the nm_tcp_socket structure
The structure is appended onto tcpSockets list (to be a candidate for select, if certain extra requirements are met)

Let’s look at how the async_read function then uses this strucuture:

void async_read(struct np_tcp_socket* sock, void* buffer, size_t bufferSize, size_t* readLength, struct np_completion_event* completionEvent)
{
    if (sock->aborted) {
        np_completion_event_resolve(completionEvent, NABTO_EC_ABORTED);
        return;
    }
    if (sock->read.completionEvent != NULL) {
        np_completion_event_resolve(completionEvent, NABTO_EC_OPERATION_IN_PROGRESS);
        return;
    }
    sock->read.buffer = buffer;
    sock->read.bufferSize = bufferSize;
    sock->read.readLength = readLength;
    sock->read.completionEvent = completionEvent;
    nm_select_unix_notify(sock->selectCtx);
}

The most important line to notice is sock->read.completionEvent = completionEvent; which will be better understood by looking at how to module is building the fdset for select:

void nm_select_unix_tcp_build_fd_sets(struct nm_select_unix* ctx)
{
    struct np_tcp_socket* s;
    NN_LLIST_FOREACH(s, &ctx->tcpSockets)
    {
        if (s->read.completionEvent != NULL) {
            FD_SET(s->fd, &ctx->readFds);
            ctx->maxReadFd = NP_MAX(ctx->maxReadFd, s->fd);
        }
        if (s->write.completionEvent != NULL || s->connect.completionEvent != NULL) {
            FD_SET(s->fd, &ctx->writeFds);
            ctx->maxWriteFd = NP_MAX(ctx->maxWriteFd, s->fd);
        }
    }
}

So by setting the read.completionEvent pointer, now the socket will be present in the FD fileset for select (readFds).

At some point (an execise for the reader to look into), select has been called and returned with something to read for the system. This will then contribute to a call to the nm_select_unix_tcp_handle_select function.

void nm_select_unix_tcp_handle_select(struct nm_select_unix* ctx, int nfds)
{
    struct np_tcp_socket* s;
    NN_LLIST_FOREACH(s, &ctx->tcpSockets)
    {
        if (FD_ISSET(s->fd, &ctx->readFds)) {
            tcp_do_read(s);
        }
        if (FD_ISSET(s->fd, &ctx->writeFds)) {
            if (s->connect.completionEvent) {
                is_connected(s);
            }
            if (s->write.completionEvent) {
                tcp_do_write(s);
            }
        }
    }
}

Here the tcp_do_read(s) will be triggered and if we follow this:

void tcp_do_read(struct np_tcp_socket* sock)
{
    if (sock->read.completionEvent == NULL) {
        return;
    }

    np_error_code ec = tcp_do_read_ec(sock);

    if (ec != NABTO_EC_AGAIN) {
        struct np_completion_event* ev = sock->read.completionEvent;
        sock->read.completionEvent = NULL;
        np_completion_event_resolve(ev, ec);
    }
}

It is obvious that the completion event will be resolved once something is available and read on the socket.

It should be possible to port the unix select implementation to other systems that offers threads and select functionallity. An example is the Nabto5 ESP-IDF implementation which is a platform for the ESP32 WIFI modules.