These processes are responsible for interacting with the administrator to start up the daemon processes, configure them, and maintain the topology of the cluster. Multiple processes are necessary to implement a quorum-based protocol for managing the cluster. The manager processes are responsible for knowing the health status of each daemon instance, known as a node. Managers send commands to bring daemons in and out of service, tell them when to interact with clients, and trigger jobs to maintain data integrity as cluster topology changes. Cluster reconfiguration involving dynamically adding and removing capacity, and adding and removing redundant data replicas, will be managed by these processes.

The current capabilities of the InfiniSQL Manager is to launch any number of configured daemon nodes and then exit. Dynamic cluster configuration, health checking, quorum management, and so forth, have not been implemented yet. The framework has been started, and progress will continue.

Daemons are the processes to which clients connect to manipulate data. Each daemon stores a portion of the database, and has access to data spread across all nodes. There is no need to shard InfiniSQL--all clients have access to the same data, regardless of node to which they connect.

The actors in InfiniSQL are implemented as POSIX threads. Classic actors are short-lived processes, such as implemented in Erlang. Applications create as many actors as necessary to perform discrete tasks, and then dispose of them. However, since it is relatively expensive in terms of processing to create POSIX threads, InfiniSQL does not follow the classic model in that regard. Instead, InfiniSQL's actors are long living threads. Each actor thread is dedicated to a particular set of functions, and each handles a very large number of concurrent activities. InfiniSQL's actor types are as follows:

For any given activity, such as processing an SQL transaction, multiple actors are involved. Each actor performs its assigned task, including sending messages to other actors. This is in contrast to traditional database design where a single worker process (or thread, depending on the particular implementation) handles all activity for any given operation. Here is a simplified flow through various actors for an example query:

InfiniSQL actor processing is event-based. An actor receives an event, works on the activity requested, stores the state of the activity, if necessary, and then returns to the event loop to wait for another event. In this way a single actor thread can handle any number of concurrent transaction activities. In the traditional worker model, each worker handles a single transaction at a time and performs all activities associated with the transaction. With actors, each actor only performs a subset of tasks for each transaction, but handles tasks for any number of concurrent transactions.

InfiniSQL messaging is implemented in the Mbox*, Message*, Outbound Gateway, and Inbound Gateway classes. Each actor has a mailbox which is an instance of Mbox. The mailbox is a multi-producer, single-consumer queue (many senders, one receiver). The queue is very simple: it is a straight FIFO. There is no prioritization of messages, no peeking at them for later consumption out of order, or any other features beyond simply putting or grabbing the next message, if available. This simplicity pays dividends in performance.

Producing and consuming messages is synchronized among multiple threads by lock free atomic operations. This is much faster than a mutex-based implementation. Internal messaging is a matter of passing pointers to objects of various types of Messages. A very important dependency is Lockless, Inc.'s memory allocator. This is a replacement for malloc(), free() and realloc(). InfiniSQL can create, send, receive and then destroy millions of messages per second per host. Only Lockless' allocator has been found to support this workload.

Message passing between nodes is handled by Outbound Gateway and Inbound Gateway actors. Messages bound for remote nodes are sent through Outbound Gateway-Inbound Gateway pairs. Sending actors serialize messages bound for remote nodes, then pass them to an Outbound Gateway. The Outbound Gateway compresses these messages using LZ4 and sends them out in batches to each remote node. Inbound Gateway decompresses the batch and sends each serialized message to its destination actor. The destination actor then deserializes and acts upon the message. Each node can have multiple gateways of each type to avoid bottlnecks.

Each actor is uniquely represented by the node upon which it resides and by the actor instance in that particular node. Each node in InfiniSQL is a process, and not a physical (or virtual host). There can be many nodes per host. Each nodeid is unique, and all actors are assigned an actorid unique within that particular node. This address is represented by the Topology::addressStruct structure.

Messages between actors within a node are sent via the MboxProducer::send() method, and received via Mbox::receive(). Each producer object is associated with a single destination Mbox object. A higher level layer for sending messages is provided by the Mboxes class. An instance of this type contains a collection of producers organized in various ways, such as by actor type (Transaction Agent, Engine, etc) and by data partition. Each actor has its own Mboxes object, which it updates upon receipt of messages from the Topology Manager. Mboxes methods allow for sending of messages based on type of destination actor and partition number. For example, Mboxes::toPartition() sends a message to the Engine responsible for a specific partition. If that resides on a remote node, then the message is passed to an Outbound Gateway. Otherwise, it's sent locally via MboxProducer::send().

There are a variety of Message types, but each inherit from the base Message class. Each message contains fields specific to the purpose of the Message, such as MessageSocket to notify a Transaction Agent that one of its clients has sent some data, or for MessageTransaction for communication between Transaction Agents and Engines. Messages are serialized and deserialized for transport between nodes. Sending actors serialize messages bound for remote hosts. Receiving actors deserialize messages from remote nodes.

Most actors' event loop is focussed around consuming messages from Mbox. Exceptions to this are the Listener, which receives socket events via epoll(), Inbound Gateway, which uses poll() for socket events, and Topology Manager, which acts upon a 0mq socket.

The transaction processing algorithm is a variation on that of two-phase locking (2PL). MVCC was considered initially, but is not scalable enough for the goals of InfiniSQL.^[1] Instead, InfiniSQL uses 2PL which incorporates some additional elements:

Most transaction processing logic is represented in the Transaction and SubTransaction classes. Transaction objects are associated with Transaction Agents, and SubTransactions are tied to Engines. Each transaction may involve records from any number of engines from any nodes. All engine activities on behalf of a transaction are performed by a corresponding SubTransaction. Each transaction, therefore, has a single SubTransaction associated with each Engine that contains records involved in the transaction. Each SubTransaction is uniquely associated with a single transaction.

Each stage of a transaction is handled asynchronously. Each Transaction Agent keeps a map of all associated transactions, keyed by transaction identifier. Each Transaction Agent maintains its own list of transactionids. Every unique transaction is identified by Transaction Agent instance and transactionid. Correspondingly, each engine keeps a SubTransaction map for each associated SubTransaction. Transaction stages are as follows:

Two-phase locking is subject to deadlocks. The Deadlock Manager actor implements a wait-for graph to resolve deadlocks. This area is under development. So, for the time being, all locked records result in a rolled back transaction, and a "lock not available" error reply to the client. This is only temporary, until lock management is properly implemented.

InfiniSQL makes extensive use of continuations. Transaction Agent activities are broken into several steps which wait for other actors, such as Engines, to complete necessary tasks. Each Transaction Agent could be handling many thousdands of transactions at the same time. Therefore, they cannot block while waiting for other tasks to complete. Instead, continuations are used to allow Transaction Agent functions to pick up exactly where they left off when replies arrive.

In conjunction with messaging and continuations, InfiniSQL implements a form of inter-actor remote procedure calls (RPC). Transaction Agents are clients, and Engines, Deadlock Manager and User Schema Manager are RPC servers. Messages from the Transaction Agent invoke a function on the "server." Function name and parameters are passed as members of the Message. Function execution populates the return message. When received, the Transaction Agent continues the function based on its state.

SQL processing goes through the following steps: