{"id":2341,"date":"2013-11-30T15:25:32","date_gmt":"2013-11-30T20:25:32","guid":{"rendered":"http:\/\/hstore.cs.brown.edu\/?page_id=2341"},"modified":"2013-12-01T22:54:36","modified_gmt":"2013-12-02T03:54:36","slug":"new-system-statistics","status":"publish","type":"page","link":"https:\/\/hstore.cs.brown.edu\/documentation\/development\/new-system-statistics\/","title":{"rendered":"Adding New System Statistics"},"content":{"rendered":"

\u00ab<\/B> Testing H-Store<\/a><\/div>
Writing New Benchmarks<\/a> \u00bb<\/B><\/div>
<\/div><\/p>\n

The following documentation describes how to collection new internal statistics from the run time system. H-Store uses a built-in stats management system that is exposed to the outside world using the @Statistics<\/a> system procedure.<\/p>\n

For the purpose of this documentation, we will refer to the component in the DBMS that collects the statistics for a specific aspect of the system as a StatsSource<\/tt>. The sub-system that aggregates multiple StatsSource<\/tt> together (either within a single partition, or at a site site), is known as a StatsAgent<\/tt>. The StatsAgents<\/tt> can be used to coalesce data in a lazy manner without needing to retrieve information directly from the StatsSource<\/tt> each time. All of the StatsAgents<\/tt> are already created in the system for you. If you are reading this documentation because yu want to collect additional information about the DBMS, then you are trying to add a new StatsSource<\/tt>.<\/p>\n

There are several considerations that one must make before adding a new StatsSource<\/tt>:<\/p>\n

    \n
  1. Where in the system is the stats data being generated?\n
  2. How often should you collect this stats data?\n
  3. Should the stats always be collected or on demand?\n<\/ol>\n

    There are two areas in the system where you can collect stats information: (1) the Java front-end layer and (2) the C++ execution engine layer. We will now describe how to add new agents for each of these.<\/p>\n

    <\/a><\/p>\n

    Java Stats Collection<\/h2>\n

    The first step is to create a new class that extends the StatsSource<\/a> base class. You will then need to hook your new source into the proper component of the overall system. There is no right\/wrong way to do this, since it depends on what data you want to collect. In general, it is best to avoid a centralized agent if multiple threads are going to be updating it. For example, if there is information that you want to collect from the PartitionExecutor<\/a><\/span>, then it is best to have a separate source per partition and then aggregate their results only when needed. If the information is global (e.g., the amount memory used by the JVM), then it is sufficient to just use a single source (since multiple threads will not need to write to it).<\/p>\n

    After you have created your new source class, you will then need to have it register with the StatsAgent<\/tt> at runtime. To do this, you need to create a new entry in the SysProcSelector<\/a> enum. As an example, let create a new stats source called “TigerStyle” that retrieves information for a single partition. We will want to create our new entry in SysProcSelector<\/tt> like so:<\/p>\n\n

    public<\/span> enum<\/span> SysProcSelector {<\/span>\n    ...\n    TIGER_STYLE<\/span> \/\/ The amount of tiger style in the system!<\/span>\n}<\/span><\/pre><\/td><\/tr><\/table><\/div>\n\n
    You then must register the source with the HStoreSite<\/a><\/span>‘s global StatsAgent<\/tt>. We will then get the StatsAgent<\/tt> handle from the HStoreSite<\/a><\/span> and register our source:<\/p>\n\n
    TigerStyleSource tss =<\/span> new<\/span> TigerStyleSource(<\/span>partitionId)<\/span>;<\/span>\nhstore_site.getStatsAgent<\/span>(<\/span>)<\/span>\n           .registerStatsSource<\/span>(<\/span>SysProcSelector.TIGER_STYLE<\/span>, partitionId, tss)<\/span>;<\/span><\/pre><\/td><\/tr><\/table><\/div>\n\n
    The last step is to hook the new SysProcSelector<\/tt> into the @Statistics<\/a> sysproc so the system knows where to get the information that you need at runtime. You must first add new FragmentId entries into SysProcFragmentId<\/a>. There will already be existing entries for other stats sources, so you just need to append yours to that list. Make sure that you use a unique id. See the documentation on adding new sysprocs<\/a> for additional information.<\/p>\n
    Next, in the Statistics<\/a> class, you add in the new FragmentIds that you just created to the system. Use the addStatsFragments()<\/tt> helper method to automatically configure everything. This is make it so that the stats request is sent to every partition and then output from each partition is returned back. Be sure to also register the FragmentIds at runtime in initImpl()<\/tt>. Finally, you will need to add the logic to retrieve your data from each partition’s stats source in executePlanFragment()<\/tt>.<\/p>\n
    <\/a><\/p>\n
    C++ Stats Collection<\/h2>\n
    For collecting stats from H-Sore’s , all of the infrastructure for copying the data from C++ into Java is in place. You just need to implement a new class that extends StatsSource<\/a>. To do this, add a new entry to the StatisticsSelectorType<\/a> in types.h<\/tt> and to the SysProcSelector<\/a> in Java. It is important that the value of these two entries matches, otherwise you will be unable to connect them together.<\/p>\n
    For example, suppose that the SysProcSelector.TIGER_STYLE<\/tt> entry from above is the 21st element in the SysProcSelector<\/tt> enum. That means in the C++ StatisticsSelectorType<\/tt> enum you need to make sure that the entry also has index offset of 21:<\/p>\n\n
    enum<\/span> StatisticsSelectorType {<\/span>\n    ...\n    STATISTICS_SELECTOR_TYPE_TIGER_STYLE<\/span> =<\/span> 21<\/span> \/\/ The amount of tiger style in the system!<\/span>\n}<\/span>;<\/span><\/pre><\/td><\/tr><\/table><\/div>\n\n
    Next, you need to register your new StatsSource<\/tt> with the EE’s StatsAgent<\/a>. This is done in the main class for the execution engine<\/a> through StatsAgent::registerStatsSource()<\/tt>.<\/p>\n
    Finally, you need to process a stats request in the EE that arrives from the Java layer. There is a single method VoltDBEngine::getStats()<\/tt> that is responsible for retrieving data from the StatsAgent<\/tt> and stores them into a VoltTable<\/tt>. In this method, you will see a switch<\/tt> statement that processes the “selector” variable passed in from Java. This flag corresponds to the StatisticsSelectorType<\/tt>. Thus, you will need to add a case statement to handle the new selector flag that you added.<\/p>\n
    <\/a><\/p>\n
    Retrieving Statistics<\/h2>\n
    To be written…<\/i><\/p>\n
    \u00ab<\/B> Testing H-Store<\/a><\/div> 
    Writing New Benchmarks<\/a> \u00bb<\/B><\/div>
    <\/div><\/p>\n","protected":false},"excerpt":{"rendered":"
    The following documentation describes how to collection new internal statistics from the run time system. H-Store uses a built-in stats management system that is exposed to the outside world using the @Statistics system procedure. For the purpose of this documentation, we will refer to the component in the DBMS that collects the statistics for a […]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":616,"menu_order":10,"comment_status":"closed","ping_status":"open","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/hstore.cs.brown.edu\/wp-json\/wp\/v2\/pages\/2341"}],"collection":[{"href":"https:\/\/hstore.cs.brown.edu\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/hstore.cs.brown.edu\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/hstore.cs.brown.edu\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hstore.cs.brown.edu\/wp-json\/wp\/v2\/comments?post=2341"}],"version-history":[{"count":11,"href":"https:\/\/hstore.cs.brown.edu\/wp-json\/wp\/v2\/pages\/2341\/revisions"}],"predecessor-version":[{"id":2356,"href":"https:\/\/hstore.cs.brown.edu\/wp-json\/wp\/v2\/pages\/2341\/revisions\/2356"}],"up":[{"embeddable":true,"href":"https:\/\/hstore.cs.brown.edu\/wp-json\/wp\/v2\/pages\/616"}],"wp:attachment":[{"href":"https:\/\/hstore.cs.brown.edu\/wp-json\/wp\/v2\/media?parent=2341"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}