Documentation > Development > Writing New Benchmarks

These instructions are for those that wish to write a new benchmark for the H-Store system. Please note that the H-Store distribution already includes a wide variety of OLTP benchmarks that work out of the box.

Throughout these instructions, we will refer to an example benchmark called the “ABC Benchmark”. We will use the project identifier “abc“. The following steps are necessary in order to write a new benchmark to use with the hstore-prepare and hstore-benchmark commands to automatically deploy H-Store on a cluster, load the data, and execute a workload using H-Store’s built-in benchmark framework.

In general, the high-level steps are as follows:

  1. Create a new directory structure that will contain your benchmark.
  2. Add your benchmark to the internal list of supported benchmarks for testing.
  3. Write the DDL file that defines your benchmark’s database.
  4. Write the stored procedure implementations.
  5. Write the ProjectBuilder class that defines properties about your benchmark.
  6. Create the Data Loader to populate the database with initial data.
  7. Create the Client Driver that will invoke transactions.
  8. Create the benchmark target configuration file.

Setup

Create a new directory in the source tree under the src/benchmarks directory that will contain your benchmark. The high-level directory for your benchmark should contain a separate directory for the stored procedures. In the example shown below, we are creating a new “abc” benchmark under the “com.example.benchmark” package:

mkdir -p $HSTORE_HOME/src/benchmarks/com/example/benchmark/abc
mkdir $HSTORE_HOME/src/benchmarks/com/example/benchmark/abc/procedures

ProjectTypes Addition

If you want to automatically build and load your benchmark project from within JUnit (using BaseTestCase) you will need to add your benchmark identifier to ProjectType.

public enum ProjectType {
    // "Benchmark Identifier" ("Benchmark Name", "Benchmark Package")
    ABC ("ABC Benchmark", "com.example.benchmark.abc");
    ...
}

Schema File

Next, create the benchmark DDL file and store it in your benchmark directory. See the VoltDB documentation on supported DDL SQL. The file must be named with your benchmark identifier followed by “-ddl.sql” For example, our example “abc” benchmark would contain the file abc-ddl.sql with the following contents:

CREATE TABLE TABLEA (
   A_ID     BIGINT NOT NULL,
   A_VALUE  VARCHAR(64),
   PRIMARY KEY (A_ID)
);
CREATE TABLE TABLEB (
   B_ID     BIGINT NOT NULL,
   B_A_ID   BIGINT NOT NULL REFERENCES TABLEA (A_ID),
   B_VALUE  VARCHAR(64),
   PRIMARY KEY (B_ID, B_A_ID)
);

Although the DBMS does not enforce foreign key constraints, it is still recommended that you define them in your DDL file so they can be used in the various automatic database design tools included with H-Store.

Stored Procedures

By design, H-Store only executes Java-based stored procedures. Each stored procedure is a separate Java class that extends the VoltProcedure API. It must specify the pre-defined queries and a run() method that is invoked for each transaction. Once again, H-Store supports the same SQL syntax as VoltDB as well as the VoltProcedure API. Each stored procedure should return an array of VoltTables. If you do not plan on using the H-Store automatic database designer or transaction estimator, you must also include the VoltDB @ProcInfo annotation information for each stored procedure.

In the example below, we define a new stored procedure called GetData that includes a single query GetA:

package com.example.benchmark.abc.procedures;
import org.voltdb.*;
 
public class GetData extends VoltProcedure {
 
    public final SQLStmt GetA = new SQLStmt("SELECT * FROM TABLEA WHERE A_ID = ? ");
 
    public VoltTable[] run(long a_id) {
        voltQueueSQL(GetA, a_id);
        return (voltExecuteSQL());
    }   
}

You can also create single-statement stored procedures without needing to create a separate Java file. These are special procedures that will automatically execute a single query and return the results. The input parameters of that query are automatically defined as the input parameters to the stored procedure. Single-statement procedures are defined in the benchmark’s project builder.

Project Builder

You will next need to create a new project builder class that extends the AbstractProjectBuilder class. A project builder defines (1) the benchmark’s Data Loader class, (2) the benchmark’s Client Driver class, (3) the default stored procedures included in your benchmark, and (4) the default partitioning scheme for your benchmark’s database. The list of stored procedure classes are simply passed into the constructor. Likewise, a list of of tables (each with their corresponding partitioning column) is passed to the constructor. Any table not included in this list will be replicated in every partition.

The Data Loader and Client Driver class files must be defined in two static parameters m_clientClass and m_loaderClass, respectively.

In the example below, the ABCProjectBuilder specifies that the data loader is ABCClient.class and ABCLoader.class. There is also one stored procedure in the benchmark and that the two tables from abc-ddl.sql are partitioned on A_ID and B_A_ID respectively. It also defines a single-statement stored procedure called DeleteData.

package com.example.benchmark.abc;
 
import edu.brown.benchmark.AbstractProjectBuilder;
import edu.brown.api.BenchmarkComponent;
import com.example.benchmark.abc.procedures.*;
 
public class ABCProjectBuilder extends AbstractProjectBuilder {
 
    // REQUIRED: Retrieved via reflection by BenchmarkController
    public static final Class<? extends BenchmarkComponent> m_clientClass = ABCClient.class;
 
    // REQUIRED: Retrieved via reflection by BenchmarkController
    public static final Class<? extends BenchmarkComponent> m_loaderClass = ABCLoader.class;
 
    public static final Class<?> PROCEDURES[] = new Class<?>[] {
        GetData.class,
    };
    public static final String PARTITIONING[][] = new String[][] {
        // { "TABLE NAME", "PARTITIONING COLUMN NAME" }
        {"TABLEA", "A_ID"},
        {"TABLEB", "B_A_ID"},
    };
 
    public ABCProjectBuilder() {
        super("abc", ABCProjectBuilder.class, PROCEDURES, PARTITIONING);
 
        // Create a single-statement stored procedure named 'DeleteData'
        addStmtProcedure("DeleteData", "DELETE FROM TABLEA WHERE A_ID < ?");
    }
}

Data Loader

The data loader is used to populate the database with some initial data before the benchmark begins. It will execute on the same node as where the BenchmarkController is executing (i.e., the same node as where the hstore-benchmark command was invoked). It can be multi-threaded, but that is left up to the implementation. The data loader needs to extend the BenchmarkComponent class and implement several methods. The BenchmarkController will invoke the main execution method runLoop() for the loader. This is where you will need to create VoltTable handles for each of your benchmark’s tables and populate them with tuples/rows. You can retrieve a Client connection to the database cluster using the BenchmarkComponent.getClientHandle() method. One also has to implement the getTransactionDisplayNames() method but this can return null.

In the example below, the data loading portion of the ABCLoader would be included in the the runLoop() method. This data is pushed to the database by invoking the @LoadMultipartitionTable procedure and passing the populated VoltTable as an input parameter. H-Store will automatically partition the table’s contents and store the tuples at the proper partitions.

package com.example.benchmark.abc
 
import org.voltdb.catalog.*;
import org.voltdb.client.Client;
import edu.brown.api.Loader;
import edu.brown.catalog.CatalogUtil;
 
public class ABCLoader extends Loader {
 
    public static void main(String args[]) throws Exception {
        BenchmarkComponent.main(ABCLoader.class, args, true);
    }
 
    public ABCLoader(String[] args) {
        super(args);
        for (String key : m_extraParams.keySet()) {
            // TODO: Retrieve extra configuration parameters
        } // FOR
    }
 
    @Override
    public void load() {
        // The catalog contains all the information about the database (e.g., tables, columns, indexes)
        // It is loaded from the benchmark's project JAR file
        Catalog catalog = this.getCatalog();
 
        // Iterate over all of the Table handles in the catalog and generate
        // tuples to upload into the database
        for (Table catalog_tbl : CatalogUtil.getDatabase(catalog).getTables()) {
            // TODO: Create an empty VoltTable handle and then populate it in batches to 
            //       be sent to the DBMS
            VoltTable table = CatalogUtil.getVoltTable(catalog_tbl);
 
            // Invoke the BenchmarkComponent's data loading method
            // This will upload the contents of the VoltTable into the DBMS cluster
            this.loadVoltTable(catalog_tbl.getName(), table);
        } // FOR
    }
}

Client Driver

After the Data Loader is finished, the BenchmarkController will then deploy one or more Client Driver instances on the client hosts. Each
The underlying BenchmarkController framework will automatically setup the client connections to each of the H-Store nodes and load the benchmark catalog.

package com.example.benchmark.abc
 
import java.io.IOException;
import java.util.Random;
import org.voltdb.client.*;
import edu.brown.api.BenchmarkComponent;
 
public class ABCClient extends BenchmarkComponent {
 
    public static void main(String args[]) {
        BenchmarkComponent.main(ABCClient.class, args, false);
    }
 
    public ABCClient(String[] args) {
        super(args);
        for (String key : m_extraParams.keySet()) {
            // TODO: Retrieve extra configuration parameters
        } // FOR
    }
 
    @Override
    public void runLoop() {
        try {
            Client client = this.getClientHandle();
            Random rand = new Random();
            while (true) {
                // Select a random transaction to execute and generate its input parameters
                // The procedure index (procIdx) needs to the same as the array of procedure
                // names returned by getTransactionDisplayNames()
                int procIdx = rand.nextInt(ABCProjectBuilder.PROCEDURES.length);
                String procName = ABCProjectBuilder.PROCEDURES[procIdx].getSimpleName();
                Object procParams[] = null; // TODO
 
                // Create a new Callback handle that will be executed when the transaction completes
                Callback callback = new Callback(procIdx);
 
                // Invoke the stored procedure through the client handle. This is non-blocking
                client.callProcedure(callback, procName, procIdx);
 
                // Check whether all the nodes are backed-up and this client should block
                // before sending new requests. 
                client.backpressureBarrier();
            } // WHILE
        } catch (NoConnectionsException e) {
            // Client has no clean mechanism for terminating with the DB.
            return;
        } catch (InterruptedException e) {
            e.printStackTrace();
        } catch (IOException e) {
            // At shutdown an IOException is thrown for every connection to
            // the DB that is lost Ignore the exception here in order to not
            // get spammed, but will miss lost connections at runtime
        }
    }
 
    private class Callback implements ProcedureCallback {
        private final int idx;
 
        public Callback(int idx) {
            this.idx = idx;
        }
        @Override
        public void clientCallback(ClientResponse clientResponse) {
            // Increment the BenchmarkComponent's internal counter on the
            // number of transactions that have been completed
            incrementTransactionCounter(this.idx);
        }
    } // END CLASS
 
    @Override
    public String[] getTransactionDisplayNames() {
        // Return an array of transaction names
        String procNames[] = new String[ABCProjectBuilder.PROCEDURES.length];
        for (int i = 0; i < procNames.length; i++) {
            procNames[i] = ABCProjectBuilder.PROCEDURES[i].getSimpleName();
        }
        return (procNames);
    }
}

Benchmark Specification File

Lastly, you will need to add a benchmark specification file to $HSTORE_HOME/properties/benchmarks/. By default, H-Store will look for the specification file in this directory based on the value of the project parameter.

This file needs to contain the following information:

Variable Name Description
builder The canonical path to the ProjectBuilder class. For example, the builder parameter for the TM1 benchmark is edu.brown.benchmark.tm1.TM1ProjectBuilder.
workload.ignore Optional comma separated list of the names of procedures from the benchmark that H-Store should exclude from transaction trace logs