tag:blogger.com,1999:blog-3920070533374606832024-03-14T00:02:43.776-07:00boomphistoyou <i>can</i> teach an old maestro a new tunestoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-392007053337460683.post-64556589561075713882013-07-26T05:14:00.000-07:002013-07-26T05:15:45.456-07:00Writing Testable ArcObjects Applications - Part 3 - Unit Test Structure<span style="font-family: Trebuchet MS, sans-serif;">In the <a href="http://boomphisto.blogspot.com.au/2013/07/writing-testable-arcobjects.html" target="_blank">previous post</a> I described the concept of inversion of control and how to decouple classes from their dependencies using interfaces. The objective of doing this is to make the business logic more testable, which now brings us to testing. However, before I jump into mocking ArcObjects code, in this post I will discuss the structure of tests, which will give some context to the unit tests I will discuss in the next topic.</span><br />
<br />
<a name='more'></a><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><span style="font-size: large;"><b>TDD vs BDD</b></span></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The main focus of this post is to describe the unit test class structures I use, which are a mixture between TDD (Test Driven Development), and BDD (Behaviour Driven Development) patterns. The difference between TDD and BDD is quite subtle, but once realised it is quite profound. Matt Wynne tackles the subject in a <a href="http://blog.mattwynne.net/2012/11/20/tdd-vs-bdd" target="_blank">TDD vs BDD blog post</a> describing TDD as very similar to BDD, except that BDD had a much more business, or customer centric focus, while TDD was more system oriented. In Matt's book - <a href="http://pragprog.com/book/hwcuc/the-cucumber-book" target="_blank">The Cucumber Book</a>, he describes unit tests as focusing on <i>"building the system right"</i>, whilst acceptance tests focus on <i>"building the right system"</i>, and we can draw a similar comparison between TDD and BDD. Since we can mock our to dependencies to test complex business logic, we can write our BDD style acceptance tests as unit test classes.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Both TDD and BDD are about -</span><br />
<br />
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">defining your specification first, i.e. the test</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">writing enough code to allow the test to fail meaningfully</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">writing the functional code to deliver the working solution for the test</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">refactoring the solution to improve the code, i.e. DRY it up (DRY - <b>D</b>on't <b>R</b>epeat <b>Y</b>ourself)</span></li>
</ul>
<br />
<span style="font-family: 'Trebuchet MS', sans-serif;">My main focus on TDD and BDD is related to the patterns of the test code.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif; font-size: large;"><b>Arrange Act Assert</b></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The pattern typically used in TDD unit tests is called the AAA pattern or <i>"Arrange Act Assert"</i>, where basically you -</span><br />
<br />
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Arrange </b>the state of the test item</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Perform an <b>act</b>ion on the test item</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><b>Assert</b> the result of the action</span></li>
</ul>
<br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">I use the AAA pattern for performing simple unit tests on self contained classes, which I can easily supply inputs, execute a test, and then verify the results.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The structure of these tests is very simple, typically a test method for each test, and perform the Arrange, Act, Assert functionality within the single test method. See the example below related to the "happy hour" example from the <a href="http://boomphisto.blogspot.com.au/2013/06/writing-testable-arcobjects.html" target="_blank">first post</a> I published on testing</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The test example below is simply checking that the <i>LookupPrice </i>method returns the expected result for the "Little Creatures Pale Ale" beer.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/stoolrossa/6087688.js"></script></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif; font-size: large;"><b>Gherkin</b></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Gherkin is a language for writing acceptance tests such that a non technical user can understand the specification, and also allows a developer to interpret what system tests will verify that the logic meets the test expectations.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Gherkin is structured as - </span><br />
<br />
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><i>Given </i>some pre condition</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><i>When </i>a particular action is performed</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"><i>Then </i>an expected result should be obtained</span></li>
</ul>
<br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">As you can see, the pattern of the test is very similar to AAA, but subtly, the test is written in relation to the expected behaviour of the system from a business logic perspective, rather than the technical execution of a single class which is being unit tested.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">From this perspective, I typically use BDD styled tests for testing business logic defined as user stories in an agile project, but for simple unit test cases on internal classes, I write AAA styled tests.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In the examples shown below, I have a base class called <i>SpecificationContext </i>that executes the <i>Given </i>and <i>When </i>methods in the test initialise. Each test class executes the Then methods as each individual test.</span><br />
<br />
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/stoolrossa/6087691.js"></script></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">I derive from the <i>SpecificationContext</i> class to provide a <i>DrinkOrderContext </i>class which creates a context that I can perform a number of different BDD styled tests on.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/stoolrossa/6087703.js"></script></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">My test classes are named according to the <i>Given</i> and <i>When</i> aspect of the test, and the test methods represent the <i>Then</i> component of the test, such that the results of running the tests show the class name in one result column, and the test in the next column. If my tests had multiple expectations i.e. Then blah And blah, then each expectation would be another test <i>Then</i>. Each test is then run independently, so there are no tests dependent on another test having been executed.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">
<script src="https://gist.github.com/stoolrossa/6087720.js"></script>
<br /><br />
<script src="https://gist.github.com/stoolrossa/6087713.js"></script>
</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Example results are shown below.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-BEDr9WvdlNQ/UfJnOMIh2TI/AAAAAAAAAM8/BGYBf1R-M3c/s1600/BDDTestCases.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="82" src="http://3.bp.blogspot.com/-BEDr9WvdlNQ/UfJnOMIh2TI/AAAAAAAAAM8/BGYBf1R-M3c/s640/BDDTestCases.png" width="640" /></a></div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif; font-size: large;"><b>Conclusion</b></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">This was just a brief introduction to the class structure of my tests so that you understand the pattern on the tests that I will be discussing in future posts. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Hopefully the discussion about TDD/BDD patterns gave you something to think about in terms of patterns for test cases. From a practical perspective, I use the Gherkin styled pattern for defining acceptance tests so that I can easily verify that I have captured a customer's expected system behaviours within my test suites, which gives me confidence that I am delivering the expected solution, as well as "safe" environment to refactor the solution to improve code reuse, or optimise for performance down the track. </span>stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com0tag:blogger.com,1999:blog-392007053337460683.post-51130857538954032812013-07-07T08:10:00.000-07:002013-07-26T05:17:14.024-07:00Writing Testable ArcObjects Applications - Part 2 - Inversion Of Control<span style="font-family: Trebuchet MS, sans-serif;">In my last blog post I introduced the subjects of Inversion of Control (IoC) and Mocking for unit testing, and this week, as promised, I will now delve into more specific detail about applying these concepts to applications written using ArcObjects.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">My focus in this article will be designing code using the SOLID principle of Dependency Inversion, and the use of Inversion of Control (IoC) frameworks for wiring applications up at runtime.</span><br />
<br />
<a name='more'></a><br />
<span style="font-family: Trebuchet MS, sans-serif; font-size: large;">Determined to be Deterministic</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">As I discussed <a href="http://boomphisto.blogspot.com.au/2013/06/writing-testable-arcobjects.html" target="_blank">last week</a>, the objective is to make our components testable using the Dependency Inversion principle, by binding our logic to abstractions (interfaces) rather than concretions. The third example I built showed how we pass our dependency instances (<i>drinksMenu </i>& <i>clock</i>) to the higher level class (<i>Checkout</i>) in the constructor. The reason we created this pattern was so that we can make the <i>CalculateCost </i>method deterministic.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">A deterministic function is a function that produces a specific result from a particular set of supplied inputs. Each combination of inputs will produce the same output every time the function is executed. In the first two examples from last week, the code was not deterministic because the results of the function were being calculated from using state that was managed internally within the class which was not part of the inputs supplied to the function. The third example was deterministic because the state being used in the calculation was supplied as constructor parameters. This means that we can verify the output of the function because we can supply all the inputs.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">You may already be familiar with deterministic functions if you have written User Defined Functions (UDF) in SQL Server. SQL Server will prevent a UDF from being created if it contains a call to getdate() within it, for the same reason that we decoupled the dependency the the DateTime variable in the example last week, a UDF requires a date to be supplied as an input to the function. The reason why UDFs must be deterministic is for memoisation so that SQL Server can cache results, but this constraint also forces developers to create testable UDFs.</span><br />
<br />
<span style="font-family: Trebuchet MS, sans-serif; font-size: large;">Guided Tour of Castle Windsor</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In my post from last week I showed some example mocks using a library called <a href="https://code.google.com/p/moq/" target="_blank">Moq</a> which I typically use for mocking in unit tests, and similarly I prefer to use <a href="http://docs.castleproject.org/Windsor.MainPage.ashx" target="_blank">Castle Windsor</a> as an IoC container. There are other libraries out there such as <a href="https://code.google.com/p/autofac/" target="_blank">Autofac</a> and <a href="http://www.ninject.org/" target="_blank">Ninject</a>, but I tend to stick to Castle Windsor, probably because I am familiar with it. Feel free to choose the library you prefer, as the concepts I am discussing are not specific to the library I am using, even though the examples themselves will use syntax that is library specific.</span><br />
<br />
<span style="font-family: Trebuchet MS, sans-serif; font-size: large;">Finally, an ArcObjects Example</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The next example will illustrate building ArcObjects code that uses an Inversion of Control (IoC) pattern. The example is relatively basic, it is a Server Object Extention which is supplied a "land use" code value in the request, which results in the service selecting the features that are defined with that land use. This is a simplistic example that illustrates the use of an IoC framework in a Server Object Extension, but in reality you would never create an extension to do this because it is achievable with a service straight out of the box.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The architecture of my example server object extension is a typical SOE class that has been autogenerated by the </span><span style="font-family: 'Trebuchet MS', sans-serif;">Visual Studio 2010 </span><span style="font-family: 'Trebuchet MS', sans-serif;">template for an SOE which comes with the ArcObjects SDK. I am building a 10.1 SOE which has class attributes that define settings used for registering the component when it is installed in the ArcGIS Server Site.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">I have defined an <i>ExampleService </i>class, with matching interface which will contain my business logic, but there isn't much business logic, just the call to the repository class, and then the packaging up of the result. The repository class, with matching interface deals with the access to the geodatabase. My call to retrieve the affected land use parcels utilises a <i>QueryFilter</i>. So my dependency chain is - <i>ExampleService</i>, which depends on an <i>ExampleRepository</i>, which depends on a <i>Workspace </i>and a <i>QueryFilter</i>.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Our dependencies will be constructed by the IoC container, and supplied to the constructor of each class, except for the <i>Workspace </i>which is being acquired from the layer info in the service itself. This is done to simplify the definition of connection details, i.e. a connection can be acquired from an existing feature class represented as a layer in the service.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: 'Trebuchet MS', sans-serif;"><span style="font-size: large;">The Castle Windsor Approach</span></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">To use the Castle Windsor IoC container, simply install the Castle Core and Castle Windsor packages using NuGet. Once installed the libraries should be referenced in your project and available to be called.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-B6356zY4C2o/Udl9qvKSGNI/AAAAAAAAAMU/IzSF3g2PLl4/s1600/NuGet.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="476" src="http://4.bp.blogspot.com/-B6356zY4C2o/Udl9qvKSGNI/AAAAAAAAAMU/IzSF3g2PLl4/s640/NuGet.png" width="640" /></a></div>
<br />
<span style="font-family: Trebuchet MS, sans-serif;">There are two main aspects to using an IoC framework -</span><br />
<br />
<ol>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Defining the mapping between interfaces and concrete types so that the IoC container can create the dependencies at run time.</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Using the IoC container to resolve the dependencies of the root object in the chain to execute the program.</span></li>
</ol>
<br />
<span style="font-family: Trebuchet MS, sans-serif;">In my example I map the interfaces/concrete types in code using an installer class. These classes implement the <i>IWindsorInstaller </i>interface which allow the IoC container to find them and derive the dependency mapping at runtime. Dependencies can also be defined in configuration files. When a developer writes a class that has dependencies they also alter the constructor of their class to supply the dependencies defined as an interface, and then add the interface/concrete type mapping to the installer file.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The second aspect of using the IoC framework is to create the instance of the root object of the program. In Castle Windsor this is explained as the <a href="http://docs.castleproject.org/Windsor.Three-Calls-Pattern.ashx" target="_blank">three calls pattern</a> -</span><br />
<br />
<ol>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Creating the Container</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Resolving the root Interface/Concretion</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Disposing the Container</span></li>
</ol>
<br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif; font-size: large;">Lets see some code....</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<script src="https://gist.github.com/stoolrossa/5943529.js"></script>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In the main SOE class <i>ExampleSoe </i>you can see that in the <i>Construct </i>method I call a method called <i>bootstrapContainer </i>to create an instance of the <i>IWindsorContainer</i>. In the <i>Shutdown</i> method I dispose of the container. Notice that I don't create an instance of the service until the operation handler is called. The <i>ExampleService </i>instance is create in the <i>createService </i>method, which executes the Resolve method for the <i>IExampleService </i>interface. When called this method will use the Installer classes to derive the concrete types from the interfaces in the constructors of each of the dependency classes up the chain. In addition to calling Resolve, the <i>createService </i>method also supplies the <i>IWorkspace </i>dependency which is held by the server object extension in one of the layer infos. The workspace dependency is passed by setting a property only, not in the constructors of the dependency chain.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<script src="https://gist.github.com/stoolrossa/5943539.js"></script><br />
<script src="https://gist.github.com/stoolrossa/5943541.js"></script><br />
<script src="https://gist.github.com/stoolrossa/5943551.js"></script><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The <i>ExampleService </i>class has a dependency on the <i>IExampleRepository </i>which it acquires in its constructor using the IoC container. The <i>GetLandUseSelection </i>method gets the landUse code passed into it in json, deserialises it and supplies it to the <i>IExampleRepository.SelectParcelsWithLandUse</i> method. Once executed it constructs a json array of the resulting parcels. Note the <i>Parcel </i>model which contains a <i>Deserialise </i>and <i>Serialise </i>method to convert to and from a json fragment.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<script src="https://gist.github.com/stoolrossa/5943544.js"></script><br />
<script src="https://gist.github.com/stoolrossa/5943547.js"></script><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The <i>ExampleRepository </i>class has a dependency on the <i>IQueryFilter </i>which it acquires in its constructor using the IoC container. The <i>SelectParcelsWithLandUse </i>validates the land use code, gets the desired feature class, executes a query using the <i>IQueryFilter </i>and converts the result cursor to a list of <i>Parcel </i>models which are returned.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<script src="https://gist.github.com/stoolrossa/5943554.js"></script><br />
<script src="https://gist.github.com/stoolrossa/5943556.js"></script><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The installer classes map the interfaces to the desired concrete types. I have separated my installers into two classes, one to handle ArcObjects dependencies and one to handle dependencies in the project itself.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">As you can see, using Castle Windsor allows us to execute the program as we typically would, without having to supply the dependencies in our constructors manually, Castle Windsor takes care of that for us, and manages caching of the types etc to improve performance.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif; font-size: large;">Conclusion</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Hopefully this gave a more concrete example of using ArcObjects with an IoC container and showed how and where the container and the root instances should be created in a server object extension project.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">There are many more features available in Castle Windsor for creating dependency instances which goes beyond my example, such as singletons and typed factories, but feel free to explore the framework further yourself.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In the coming posts I will focus more on Unit Testing and Mocking.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"> </span><br />
<div>
<br /></div>
stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com0tag:blogger.com,1999:blog-392007053337460683.post-35464161777459321702013-06-30T03:01:00.000-07:002013-07-07T08:16:19.831-07:00Writing Testable ArcObjects Applications - Part 1<span style="font-family: Trebuchet MS, sans-serif;">Having previously worked in a software product development company for a number of years, and now back working with Esri technology again, I have been a little surprised by how few internet articles there are related to Unit Testing, Mocking, and Inversion of Control for applications built with ArcObjects, so I thought it might be fun to do a multiple article series on writing testable applications using ArcObjects.</span><br />
<br />
<a name='more'></a><span style="font-family: Trebuchet MS, sans-serif;">The target audience for these posts will be GIS focused professionals who work with ArcObjects, who aren't exactly experts in using Unit Testing, Mocking, or Inversion of Control frameworks, but are interested in learning about ways to decouple business logic from dependencies, and improve test-ability generally. For all the software engineering gurus out there, you probably won't learn anything that you don't already know about.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">As I mentioned, I intend to release a number of posts over the next couple of weeks that address testing topics, but for this initial post I will only break the ice with a typical trivial contrived example, but rest assured I will be discussing specific ArcObjects testing in the coming weeks. :)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<b><span style="font-family: Trebuchet MS, sans-serif; font-size: large;">Do Me A SOLID?</span></b><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">A nice introduction to the structure of the applications I will be demonstrating in this series of articles is to reflect on the SOLID principles of object-oriented software design. These principles were described by <a href="https://sites.google.com/site/unclebobconsultingllc/" target="_blank">Uncle Bob Martin</a> quite a few years ago. The letters SOLID stand for -</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><b><span style="font-size: large;">S</span></b>ingle Responsibility - Classes should have a single focus that all their operations are aligned to.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><b><span style="font-size: large;">O</span></b>pen/Closed - Classes should be open to extension, but closed for modification.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><b><span style="font-size: large;">L</span></b>iskov Substitution - Subclasses should be able to substitute for the classes they derive from.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><b><span style="font-size: large;">I</span></b>nterface Segregation - Classes should support multiple small interfaces rather than one large one.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><b><span style="font-size: large;">D</span></b>ependency Inversion - High level classes should not depend on instances of low level dependencies, but instead depend on abstractions of those dependencies.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Generally we can see that the classes in the ArcObjects framework adhere fairly closely to the SOLID principles, and my topic of testing will focus quite a bit on the last principle - Dependency Inversion, because this is what ultimately makes our applications testable.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<b><span style="font-family: Trebuchet MS, sans-serif; font-size: large;">Decoupling Our Dependencies</span></b><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The following example illustrates an approach to decoupling your high level classes from their dependencies to make the code more testable.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In our example, we have started a bar called "The Open Door". Since we were a software developer prior to being a bar proprietor, we decide to write our own point of sale system.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In the example below you can see that we have a <i>Checkout </i>class which acquire's its own drink price list within the <i>CalculateCost </i>method. It then looks at each of the order items and looks up the drink price from the price list, and multiplies this by the quantity of drinks in the order item. The total cost then has 10% tax added.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/stoolrossa/5894100.js"></script></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">You can see that the test itself is quite fragile because it makes an assumption about the cost of the drinks to arrive at the expected total, so if the price list changes we will have a problem. One solution might be to get the price list inside the test and look-up the current prices to arrive at the expected total, but then what is to stop us from introducing a bug in this code, and is this what we are really testing? i.e. the looking up of the price? Aren't we really focused on out ability to calculate the total and the additional tax? Regardless, the code in this class can be tested but is tightly coupled to its dependency data.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">What if we introduce new business logic? Say on a Friday we want to offer half price drinks from 5pm till 6pm, so we alter the logic as follows.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/stoolrossa/5894175.js"></script></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The code we have written now is very difficult to test because we would need to execute the test at the right time to get a particular expected result. So we have introduced more fragility to our tests.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">To overcome the issues we have seen in the past two code examples we will expose our price list and date time provider (clock) as interfaces, and then pass in instances of implementations of those interfaces to our <i>Checkout</i> constructor.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/stoolrossa/5894325.js"></script></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">As you can see, the code in the <i>CalculateCost</i> method did not vary dramatically, but the main difference is that we are now passing in instances of our dependencies in the constructor. So you may ask, what was so great about making that change? Well now we have decoupled our code from our dependencies such that we are still tightly integrated using the interfaces but loosely coupled to the concrete implementations such that we could supply other instances of dependencies if we wanted to, i.e. for testing.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The example below shows our new tests which supply the price list (<i>drinksMenu</i>) and clock to the <i>Checkout </i>class.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/stoolrossa/5894375.js"></script></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Since we are mocking our dependencies it means that we don't need our entire application stack in place to perform testing on our business logic. The big benefit with this is that we can run our tests at any time and still verify all of our business logic, allowing developers to be aware of having introduced a bug into existing code, and giving them a fast feedback loop.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Something that may look strange is the use of the mocking framework to create instances of classes that implement <i>IPriceList </i>and <i>IClock </i>and the setup of mocked results to be returned from their method calls. I am using a library called <a href="https://code.google.com/p/moq/" target="_blank">Moq</a> to do this, but in reality I could have created my own class definitions that implement <i>IPriceList </i>and <i>IClock </i>and created methods that returned my expected values. The convenience of creating mocked objects is that I don't have to maintain a lot of redundant concrete implementations just for testing, and can build the mocked expectations on the fly in the tests that they apply to, for example, if we created our own implementations of <i>IClock </i>for happy hour and normal hours, I would need two separate classes, but using the Moq framework I simply define the expected output from the <i>GetDateTimeNow </i>method to be either 6:26pm on Sunday 30th of the June 2013, or 5:13pm on Friday 28th of the June 2013 to suit what I am testing.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">We will be looking at the Moq framework in a bit more depth in the coming weeks when we start talking about mocking ArcObjects interfaces, for now just picture a mocking framework as something that allows us to create fake dependency objects on the fly that implement a particular interface and produce results that we specify.</span><br />
<br />
<b><span style="font-family: Trebuchet MS, sans-serif; font-size: large;">Inversion of Control (IoC)</span></b><br /><br />
<span style="font-family: Trebuchet MS, sans-serif;">In the examples shown above we resolved the fragility (from a testing perspective) of our Checkout class by re-factoring our class to be handed instances of its dependencies in the constructor. This structure of development represents the concept of Inversion of Control, where the class doesn't create it's own instances of it's dependencies, but the code calling it does. This form of Inversion of Control is referred to as Dependency Injection, specifically as Constructor based Dependency Injection. If you have heard of Inversion of Control previously it is most likely to have been associated with articles by <a href="http://www.martinfowler.com/articles/injection.html" target="_blank">Martin Fowler</a>, who describes different forms of IoC via Dependency Injection, Factories and Service Locators.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">At this point you may be getting worried about -</span><br />
<br />
<ol>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">What happens if my class required hundreds of dependencies?</span></li>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Is this pattern opposed to the concept of encapsulation in object-oriented programming</span><span style="font-family: 'Trebuchet MS', sans-serif;"> - how will I know that the classes I am using need dependencies supplied, and what their types will be?</span></li>
</ol>
<br />
<span style="font-family: Trebuchet MS, sans-serif;">In response to the first item, if we build our application using the SOLID principles then our classes should only perform narrow bands of operations that they are responsible for, and should be built up from classes that compose their logic using other dependencies.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">In response to the second question, we will not be manually creating instances of dependencies in our own code, we will use an IoC framework to do that for us, which should provide the encapsulation we are familiar with.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif; font-size: large;"><b>Conclusion</b></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Hopefully the examples above gave you a sneak peak into the world of unit testing with mocked objects, and whet your appetite for further discussion on the topics of mocking ArcObjects interfaces. The next topic I will delve into is IoC frameworks, and how we set these up to work with ArcObjects based applications.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com0tag:blogger.com,1999:blog-392007053337460683.post-56904821013779334702013-03-12T06:16:00.001-07:002014-08-20T22:05:13.705-07:00Green Framing All The Things!<br />
<span style="font-family: Trebuchet MS, sans-serif;">This will appear to be a strange subject to post about considering my other content, but I recently went through a positive thinking reset, and I was surprised and amazed by some of the results, so I thought I would share.</span><br />
<a name='more'></a><br />
<span style="font-family: Trebuchet MS, sans-serif;">Back last year while speaking to a customer I was informed about a training program their organisation did which among other things discussed behaviour in meetings and referred to "framing things" and visualising positive approaches as "green", and to not bring closed thinking to a discussion, which would be considered "red". At the time I thought it was an interesting approach, although I didn't think too much about it because it was not a program I had been involved in.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">A few weeks later I was reading some articles about positive thinking and came across some interesting philosophies which led to me discussing some of the concepts at a local <a href="http://www.meetup.com/" target="_blank">meetup</a> group, and this discussion developed into me borrowing the "framing" concept I had heard about previously to coin the phrase "Green Framing".</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-M1A_YG_vwJ4/UT8mplQIRXI/AAAAAAAAAKE/1CbPVyhth5A/s1600/GreenFrame.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" src="http://3.bp.blogspot.com/-M1A_YG_vwJ4/UT8mplQIRXI/AAAAAAAAAKE/1CbPVyhth5A/s320/GreenFrame.jpg" height="256" width="320" /></span></a></div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif; font-size: x-small;">Caveat - I fully realise "Green Framing" is not my personal invention but in fact collection of stolen concepts, for which I have attempted to reference the original authors where possible :)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The first rule of Green Framing is easy - frame things in a positive way, just like the original meeting concept was explained to me. Don't be dismissive or negative towards anything. </span><span style="font-family: 'Trebuchet MS', sans-serif;">There is a famous quote by Henry Ford who said -</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;"> <i><b>"Whether you think you can, or you think you can't, either way you're right"</b></i></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><i><b><br /></b></i></span>
<span style="font-family: Trebuchet MS, sans-serif;">- I know, it is a bit of a cliché but when you think about it, its true, its all too easy to be pessimistic and dismissive of something rather than have the courage to face something difficult to achieve.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">One of the things I found inspiring around the time was a <a href="http://www.ted.com/talks/jane_mcgonigal_the_game_that_can_give_you_10_extra_years_of_life.html" target="_blank">talk by Jane McGonigal</a>, in which among other amazing things she also highlights the negativity bias of the brain, and how we should exercise a 3:1 ratio of positive thoughts to negative thoughts. I have more recently read an <a href="http://www.lifehacker.com.au/2013/02/rewire-your-brain-for-positivity-and-happiness-using-the-tetris-effect/" target="_blank">article on LifeHacker about rewiring the brain for positivity</a> which emphasises some of the same points raised by Jane.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Rule two of Green Framing is about identifying and measuring success in smaller increments than which you should measure problems or issues. This sounds like common-sense but I had never really thought about it, and in fact this concept is what led me to get most excited about Green Framing. Basically it is about celebrating the success you have by clearly identifying, and not becoming complacent about it. Likewise, don't be bogged down by problems, just identify them, come up with a plan to solve them, and then execute that plan. This is definitely not about ignoring problems - its just a reminder that you produce success and encounter problems on a daily basis, and that you should not fixate on the pessimistic outcomes.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The third rule, which I consider a life foundation generally is respect for others. Regardless of Green Frames or whatever, respect and politeness are paramount to social interaction, and should always exist.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The other aspects of Green Framing are borrowed from Jane's talk as mentioned above:</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Take time out for physical exercise - When I can I do some sailing or cycling on the weekend</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Take a break to do some unrelated mental exercise - Sometimes I like to do a quiz on <a href="http://www.sporcle.com/" target="_blank">Sporcle </a>at lunch time</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">Interact socially with others - enjoy time spent with your family, engage with your friends, or interact with peers - one of the things I have grown to really value over the last few years are the acquaintances (I now consider friends) gained through <a href="http://www.meetup.com/" target="_blank">meetup</a> groups around my home town. </span></li>
</ul>
<br />
<span style="font-family: Trebuchet MS, sans-serif;">So as you can see, there isn't really that much to Green Framing, it is certainly not a silver bullet to solve all your problems. In my case I use the term "Green Frame" simply as a mnemonic to remind me to excercise some of these techniques, for which it is easy to fall out of practice with. There is a blog post I sometimes </span><span style="font-family: 'Trebuchet MS', sans-serif;">refer back to by Justin Hennessy on </span><a href="http://justin-hennessy.blogspot.com.au/2012/02/agile-team-reset.html" style="font-family: 'Trebuchet MS', sans-serif;" target="_blank">An Agile Team Reset</a><span style="font-family: 'Trebuchet MS', sans-serif;"> which I like because it reminds us of the need to pause and reflect on our core values, so that we can cultivate higher value results with our core beliefs acting as a foundation. In a way this is how I see Green Framing - as a reminder of what my core values should be.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">This leads to my more recent revival of the Green Frame concept. I was recently involved in some challenging workshops at a customer site which involved a number of representatives from different parts of the business. Some of these staff had developed some friction between themselves that led to some awkward, and at times heated communication. I'm not sure where my courage came from, but after a break in proceedings I raised the concept of "Green Framing", and suggested that we each in turn should communicate to the rest of the group why we "love" the project, and I then spoke about the attributes of the project which I valued most, and what I was looking forward to in regard to future outcomes of the project. The results were amazing, because we could see that everyone had a lot of energy and belief in the project, and that it was just frustration and misguided passion that was leading to the previous negativity. The net result was that the workshop session that followed was amazingly positive, with lots of valuable outcomes.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">This was such a strange subject for me to write about, but the result after applying a Green Frame approach to clear the air in the recent meeting was so profound that it inspired me to write this post.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Sorry for all the words, I apologise in advance for any tl;drs. </span> <br />
<br />stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com0tag:blogger.com,1999:blog-392007053337460683.post-72223184094555519242012-12-13T06:53:00.000-08:002012-12-13T11:47:07.779-08:00Serving-up ArcGIS Map Tiles From Amazon S3<br />
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Its been a long time
since my last blog article, having changed jobs late last year and being in
"the zone" getting up to speed with how my new company operates, and responding to new challenges that my new role brings. However I recently had a chance to utilise
Amazon Web Services' S3 service for publishing map tile caches, and since I
wasn't able to find a great depth of information on the subject I figured I
would post about it.</span></div>
<div style="font-size: 11pt; margin: 0in;">
</div>
<a name='more'></a><span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">While I say I
couldn't find much information about publishing map tile caches in S3, I was
able to find a lot of blog posts talking about the subject, but found some of
these to be fairly general, or I struck issues that I could not find
information on resolving. In my case I
needed to publish an Esri map tile cache generated from ArcGIS Server, and
consumed by a HTML/Javascript application that utilised the ArcGIS Javascript
API.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">I found a lot of
helpful information on the subject from the following blog pages, and I thank
the authors for their information -</span></div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Trebuchet MS, sans-serif;"><a href="http://thunderheadxpler.blogspot.com.au/2008/09/arcgis-tiles-on-amazon-s3.html">Mansour Raad</a></span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Trebuchet MS, sans-serif;"><a href="http://www.azavea.com/blogs/labs/2010/02/arcgis-server-cache-in-s3/">Azavea Labs</a></span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Trebuchet MS, sans-serif;">ROK Technologies <a href="http://www.esri.com/news/arcuser/1009/cloudcomputing.html">Article </a>and
<a href="http://proceedings.esri.com/library/userconf/serug11/papers/pap/harris_leveraging_the_amazon_s3_infrastructure.pdf">Presentation</a></span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Trebuchet MS, sans-serif;">Esri <a href="http://blogs.esri.com/esri/arcgis/2012/12/05/deploying-web-mapping-apps-on-amazon-s3/">Blog Article</a> - published
after I completed my project :)</span></li>
</ul>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">As I mentioned I
found some of these articles to be a little generalised relating to the way
that the tiles were stored in S3, and in some cases how they were being
consumed. The steps below detail how I
was able to publish my map tiles.</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif; font-size: large; font-weight: bold;">Cache Generation</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Since the end result
we are seeking to achieve is to access map tiles from S3, rather than
requesting tiles from ArcGIS Server, the map tiles will need to be created as
an Exploded cache, rather than a Compact cache.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-1oURl9mL1Mo/UMnjOJpYffI/AAAAAAAAAJA/d-szl0N2DzM/s1600/CacheStorageType.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" height="400" src="http://1.bp.blogspot.com/-1oURl9mL1Mo/UMnjOJpYffI/AAAAAAAAAJA/d-szl0N2DzM/s400/CacheStorageType.png" width="388" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">The reason for this
is that an Exploded cache will create the tiles as separate image files, in my
case PNG files. A Compact cache stores
tiles in groups inside a single file called a bundle file, from which ArcGIS Server
will retrieve the requested tile. Web
applications/browsers cannot consume a bundle file directly because it is a
format utilised internally by ArcGIS Server.</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif; font-size: large; font-weight: bold;">Cache Directory Structure</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">When the cache
generation process executes the tiles will be produced in directories
under -</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">C:\arcgisserver\arcgiscache\<SERVICE_NAME>\Layers\_alllayers</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Within this
directory there will be a subdirectories which represent each scale Level of
the map tile cache. Each level is
prefixed with L, and is represented as a level integer starting from the
highest zoom scale, moving down to the most detailed zoom scale having the
largest level integer.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-XX4PX1iJCU4/UMnjeBl1fFI/AAAAAAAAAJI/Qr4MfBIyDrM/s1600/LevelsDirectories.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" height="158" src="http://1.bp.blogspot.com/-XX4PX1iJCU4/UMnjeBl1fFI/AAAAAAAAAJI/Qr4MfBIyDrM/s200/LevelsDirectories.png" width="200" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Within each Row
directory will be the map tile files themselves, represented as Columns.</span></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-FXVkMESIkyc/UMnjs1PWNYI/AAAAAAAAAJQ/mOy5joppG1c/s1600/RowsDirectories.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" height="320" src="http://2.bp.blogspot.com/-FXVkMESIkyc/UMnjs1PWNYI/AAAAAAAAAJQ/mOy5joppG1c/s320/RowsDirectories.png" width="201" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Within each Row
directory will be the map tile files themselves, represented as Columns.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/--LQkP9gzP9o/UMnj-0v24oI/AAAAAAAAAJY/6O0F22iumQA/s1600/ColumnFiles.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" height="433" src="http://4.bp.blogspot.com/--LQkP9gzP9o/UMnj-0v24oI/AAAAAAAAAJY/6O0F22iumQA/s640/ColumnFiles.png" width="640" /></span></a></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"> Something to note is
that the Row and Column integer values are represented in hexadecimal, with the
directory/file name prefixed by R and C respectively.</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif; font-size: large; font-weight: bold;">S3 Structure</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">There is a lot of
literature that describes what S3 is, especially on the AWS website, but as a
quick backgrounder (related to the way the map tiles will be stored), S3 is
basically a storage container for objects which can be accessed by a key. This differs from a typical disk storage
structure that breaks a volume up in to hierarchies of directories, and stores
files in those directories, referencing them by their path within the directory
structure.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Within S3, a
"bucket" can be created, which represent a storage area for a
collection of key/object pairs. The Esri
blog article mentioned above goes into more detail about S3 and configuring
buckets etc, so I will not reproduce that information.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">The interesting
thing about the key/object storage of S3 relating to the cache structure
described above, is that the objects stored in S3 can be keyed in such a way
that their keys represent a relative path from the bucket name, thereby
mimicking a typical disk storage structure.
It is via this capability that I maintained my map tile cache in the
same relative structure to the way it was generated by ArcGIS Server.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">To copy the cache I
used a utility called Cloudberry S3 Explorer.
This allowed me to copy a file structure on disk to my S3 bucket. In my case I copied the _alllayers directory
over into the S3 bucket.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Note that a map
cache can, and most likely will contain hundreds of thousands, or even millions
of individual map tiles, and since S3 is effectively a REST service that
provides GET, PUT, POST, DELETE actions on the bucket, to PUT each individual
file in the S3 bucket on the Amazon Cloud from a local machine will take a long
time to process, especially if the region the bucket resides in has a
relatively high network latency back to the local machine.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">A better approach
would be to either generate the map tile cache on an EC2 instance running in
the Amazon Cloud in the same region as the S3 bucket, or, if the cache needs to
be copied from a local machine, zip the cache into a single zip file, or a number
of zip files depending on the overall size of the cache, and copy the zip files
to an EC2 instance, and then move them into S3 from the EC2 instance.<span style="font-size: 11pt;"> </span></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><span style="font-size: large; font-weight: bold;">Consuming the Map Tile Cache</span><span style="font-size: 11.0pt;"> </span></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Now that the cache
has been moved to S3, we now want to consume the cache in our application logic
such that the map tiles are downloaded directly from S3 to the browser as we
pan around the map.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">To do this our
application logic needs to know the specific details of the origin, scale
levels, pixel resolution of each scale level, etc of the cache. There are a couple of ways that this can be
achieved -</span></div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Trebuchet MS, sans-serif;">Define the layer details in
the application logic</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Trebuchet MS, sans-serif;">Fake the MapServer REST
services directory json response</span></li>
</ul>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">In both these
approaches, the application logic is only able to consume the pure map tiles
published in S3. Other functionality
related to the service such as accessing the legend, through the typical REST
endpoint will not be available. </span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif; font-weight: bold;">Defining layer details in application logic</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">One way to define
the map service details for the TiledMapServiceLayer class is to define the
details within the class definition. The
ArcGIS Javascript API is based on Dojo, which provides functionality to define
javascript classes in a "classical" way similar to typical
object-oriented languages. This is done
by using the dojo.declare function to define the name of the new derived class,
the base classes that are being inherited, and the definition of the methods of
the new class.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Like languages that
use class based inheritance, the type defined in the declare function can have
a constructor which can initialise properties of the instance of the
class. In the example below you can see
the constructor method initialises the map service details such as
initialExtent, fullExtent, spatialReference, and most importantly for the tile
cache, the tileInfo, defining the levels of detail, their scales, resolution,
etc.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">The only other
method is the overridden getTileUrl method which gets the tile image url based
on the level, row, and column number passed into the method. This is covered in more detail in a section
below.</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/4276435.js"></script></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif; font-weight: bold;">Faking the MapServer REST services directory json
response</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">An interesting way
of supplying the tiled map service's details to the application logic is to
store those details in the S3 bucket along with the tiles themselves.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">A number of the blog
articles referenced above discuss this approach, where they acquire the map
service's json document using the following URL - <a href="http://%3Cserver_name%3E/arcgis/rest/services/%3cMAP_SERVICE_NAME%3e/MapServer?f=json">http://<SERVER_NAME>/arcgis/rest/services/<MAP_SERVICE_NAME>/MapServer?f=json</a></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">The response from
this request is then saved in a file called <span style="font-weight: bold;">MapServer</span>,
with no extension, and then saved to S3.
When accessed via REST, this has the effect of looking like a request to
the ArcGIS Server endpoint. The
Javascript API will apply a ?f=json query string parameter to the URL, but S3
will ignore it an pass back the file as a response.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">By default files
stored in S3 will have a Content-Type header of application/octet-stream, so
the content header should be changed to application/json so that the client
receives the response with the header it expects. To change the Content-Type header I used
CloudBerry S3 Explorer, as shown in the images below -</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-c3G6NsPFLi8/UMnkTzj5mGI/AAAAAAAAAJg/vJrKM9A_wWE/s1600/SetHttpHeaders1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" height="320" src="http://1.bp.blogspot.com/-c3G6NsPFLi8/UMnkTzj5mGI/AAAAAAAAAJg/vJrKM9A_wWE/s320/SetHttpHeaders1.png" width="249" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-IfsyWwODHmw/UMnkbA4ZRuI/AAAAAAAAAJo/C08lL-9ZD7A/s1600/SetHttpHeaders2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Trebuchet MS, sans-serif;"><img border="0" height="400" src="http://2.bp.blogspot.com/-IfsyWwODHmw/UMnkbA4ZRuI/AAAAAAAAAJo/C08lL-9ZD7A/s400/SetHttpHeaders2.png" width="397" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="font-size: 11pt; margin: 0in;">
</div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">The approach of
exposing the raw json in a file in S3 works ok if the website itself is hosted
in S3 as well, but if your website is in a different domain, then the ArcGIS
Javascript API will make cross-domain ajax request using JSONP. JSONP is an approach used to make requests to
services that don’t originate from the same site the page is published from by
padding (the "p") the JSON response with a callback function name,
and then injecting the entire response inside a script tag on the page. This means that for scenarios such as this,
the raw json is not enough to store in the MapServer file, otherwise the client
API will raise an error such as "Unexpected token :".
The source URL of the response that threw the error will give you a clue
as to how to fix the problem -</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<a href="http://%3Cs3_bucket%3E.s3.amazonaws.com/MapServer?f=json&callback=dojo.io.script.jsonp_dojoIoScript1._jsonpCallback"><span style="font-family: Trebuchet MS, sans-serif;">http://<S3_BUCKET>.s3.amazonaws.com/MapServer?f=json&callback=dojo.io.script.jsonp_dojoIoScript1._jsonpCallback</span></a></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">This indicates that
the response should look like a javascript function call to
dojo.io.script.jsonp_dojoIoScript1._jsonpCallback, passing in the raw json as
an argument, i.e.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Courier New, Courier, monospace;">dojo.io.script.jsonp_dojoIoScript1._jsonpCallback(<RAW_JSON_FROM_MAPSERVER_REQUEST>)</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Part of the problem
of defining the MapServer file like this is that the function name will be
dynamic depending on when the ajax call is made from the client. In this case Dojo is calling the script
object dojoIoScript1 because it is the first ajax call that is being made. In theory there may be many of these which
are unable to be determined until runtime, so hard-coding the function name in
the MapServer file may not help.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">In my opinion this
is more of a hack than anything else, but it was interesting to do from the
view point of seeing how it all hangs together.
In the case of my project, there was an intention to access the legend for
the service through the ArcGIS Server REST endpoint, so I came up with a hybrid
structure for the application logic which uses both ArcGIS Server for the
definition of the service and the other related endpoints, but uses S3 for
accessing the cache tiles. The hybrid
application logic is discussed further below.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">If you don't want to
access the ArcGIS Server endpoint at all, then I would recommend the most
robust approach to be defining your own service details within the application
logic, as discussed above. If you want
to abstract your service details from the logic to get the tiles from S3,
simply use a configuration structure which passes the tiled service details in
the constructor of the class i.e.</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/4276447.js"></script></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif; font-size: large; font-weight: bold;">Getting tiles from S3</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Apart from the
application logic being aware of the cache's specifications, it also needs to
override the typical operation to get tiles from the cache so that it acquires
the tiles from the S3 instead.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">To do this we
override the getTileUrl method, which is passed the level, row and column
details, and expects a URL to a tile in
return. The method is passed the level,
row and column numbers as integers, but as I mentioned earlier, the cache
created directories and files with hexadecimal values, so these integers need
to be converted to hexadecimal and have an L, R or C added to the level, row,
and column value respectively.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">This path is
appended to the root of the cache location residing in the S3 bucket, and the
extension of the image file is appended to the file name.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">The code examples
above illustrate this logic.</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif; font-weight: bold;">Hybrid Application Logic</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">The project I was
involved in needed to also access the legend for the tiled map service via a
legend widget, but due to the relocation of the cache to S3, this functionality
was not available. To get around this, our
ArcGIS Server instance was configured to retain the REST endpoint for the
cached service, but remove the physical cache tile images from the disk, and
moved these to S3.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">In my application
logic I then created a "hybrid" tiled map service layer class (shown
below) which is supplied the ArcGIS Server service url, as well as the url to
the root of the cached tiles in the S3 bucket.
This results in the layer determining its definition from ArcGIS Server,
but then accessing S3 to get the cached tiles.</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><script src="https://gist.github.com/4276415.js"></script><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif; font-size: large; font-weight: bold;">Conclusion</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">As you can see S3 is
a useful tool for storage of objects being accessed via REST, and map tiles are
a perfect candidate for this approach.
The storage of cache files on S3 appears to be more cost effective,
considering the level of availability of the S3 bucket, when compared with
achieving the equivalent with Elastic Block Storage (EBS), which is the Amazon
equivalent of virtual disk storage.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Another advantage of
S3 is that it can be paired with CloudFront, which is Amazon's content delivery
network service for delivering cached content to edge locations throughout the world,
which is useful for services which are utilised world wide.</span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Trebuchet MS, sans-serif;">Hopefully the
instructions above will help anyone who intends to expose map tiles in this
manner.</span></div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com3tag:blogger.com,1999:blog-392007053337460683.post-84819349232849692632011-10-25T10:13:00.000-07:002011-10-27T21:08:55.075-07:00Serving-up GeoJSON while having a REST on the GeoCouch<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Wow, it's been a while. Between having my tonsils removed (which is not pleasant as an adult), to finishing up with my employer of 11 years and starting a new job, things have been rather hectic over the last few months, but I have found time to put together my third post on my GeoJSON adventure that I have been dabbling with for the last 6 months or so.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I set out on this </span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">GeoJSON </span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">journey to investigate the definition as a spatial format specification, and to see what I could get for free in terms of interoperability, and I have been pleasantly surprised by the number of applications that support the format.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In my first post on <a href="http://boomphisto.blogspot.com/2011/07/simple-maps-using-geojson-and-tile5.html">Tile5 rendering GeoJSON</a> I was interested mainly on the map rendering side of things and what I needed to do to massage my data into GeoJSON to take advantage of that rendering.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In my second post on <a href="http://boomphisto.blogspot.com/2011/07/nodejs-express-leaflet-postgis-awesome.html">Leaflet rendering GeoJSON passed from ExpressJS/NodeJS</a> I delved more into the interoperability of GeoJSON between Leaflet on the client, and the javascript application server serving data from PostGIS, which was able to return query results in GeoJSON format.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In this third post I have taken the next step, storing my GeoJSON data in a NoSql database - CouchDB, and exposing it directly from the database to the client.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<a name='more'></a><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>What is CouchDB?</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><a href="http://couchdb.apache.org/">CouchDB </a>is a document database, which unlike a relational database, has no schema, allowing you to store structured objects against a key.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">CouchDB provides a RESTful api exposing data in JSON, which made me wonder whether I could use CouchDB to store <a href="http://geojson.org/geojson-spec.html">GeoJSON structured objects</a> and execute queries directly from the client, without any "middle-man" application server logic.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The interesting thing about queries in CouchDB is that records are not in a table as such, so views are defined which use a MapReduce function to define an index of documents. The view can then be filtered by values relating to the view.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In the case of my investigations, I wanted to use the same data set that I have used previously, being the cadastral layer for the state of Queensland, which represents 2.5 million land parcels over an area spanning 2200km x 1700km, so the ability to index and select small extents of records is very important, an using a tradional index is not sophisticated enough for making spatial queries.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>What is GeoCouch?</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As I investigated CouchDB further I found that there is an extension to CouchDB called <a href="http://docs.couchbase.org/couchbase-geocouch-guide/index.html">GeoCouch</a> created by <a href="http://vmx.cx/cgi-bin/blog/index.cgi">Volker Mische</a>. The purpose of GeoCouch is to provide spatial views in CouchDB, utilising a spatial index structured as an R-Tree.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This interested me a lot because it provided me with a solution for querying my spatial data.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>My Application</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As a recap, for those that have not read my <a href="http://boomphisto.blogspot.com/2011/07/simple-maps-using-geojson-and-tile5.html">first</a> or <a href="http://boomphisto.blogspot.com/2011/07/nodejs-express-leaflet-postgis-awesome.html">second</a> post on GeoJSON, the objective of my application to provide a simple mapping application with a base map layer which then overlays cadastral data which is selected by the extent of the map currently being viewed.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">To achieve this objective I wanted to use CouchDB to store my data in GeoJSON format, and serve it directly to my Leaflet based client application without having to use an application server to arrange the data retrieved from the database, as per the solution in my <a href="http://boomphisto.blogspot.com/2011/07/nodejs-express-leaflet-postgis-awesome.html">second post</a>.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Setting Up CouchDB</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The easiest way for me to install an instance of CouchDB with GeoCouch already built-in was to install <a href="http://www.couchbase.com/">Couchbase</a>. Couchbase comes in a number of flavours, from the ultra-scalable Membase Server right down to a mobile device capable version. In my case I installed the Couchbase Single Server. I downloaded and installed the 1.1.2 deb package using:</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">sudo dpkg -i couchbase-single-server-community_x86_1.1.2.deb</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In my case the service did not start after install, so I started it at the command line using:</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">sudo /etc/init.d/couchbase-server start</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">You can verify that Couchbase is installed and executing by opening the url http://127.0.0.1:5984 in a browser (on the server machine), which will respond with a welcome message and the version number in a JSON response.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b><span class="Apple-style-span" style="font-size: large;">Kicking back on the Futon</span></b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The Couchbase Single Server comes with an application called Futon which is used for managing it. Futon can be opened in a browser window with the url http://127.0.0.1:5984/_utils.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The opening screen shows the overview of the server, with a listing of the databases on the server. Initially there will only be one database - _users.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">You can create your database by clicking the Create Database link. My database was called spatial.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Futon can be used for a number of maintenance functions, including creating Views, which I will discuss more of further on.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Loading the Data</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Obviously after creating my database the next step was to load my data into Couchbase. On my same machine I had my PostGIS database that contained my cadastral dataset, so I wrote a simple application in NodeJS to load the data from PostGIS into Couchbase.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Why NodeJS? Well I wanted to have a look at the javascript frameworks that could access CouchDB, and I could reuse some of the code I used to perform queries on the PostGIS database for my second post on GeoJSON.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The library I used to access the CouchDB database was Cradle. Cradle provides basic database access for reading and saving documents, as well as a number of administrative and maintenance functions.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">My data loading applicaton steps through the PostGIS data 1000 records at a time, and saves the block of 1000 documents to the CouchDB database. Since NodeJS is asynchronous the program operates sequentially by calling the main loadPostgisRecords function recursively from the deepest closure, after Cradle performs the save of the block of 1000 documents.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
<script src="https://gist.github.com/1313139.js?file=loadcouchdata.js">
</script><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">It took around 30 minutes for my 2.5 million records to load, which isn't too bad considering it took about 20 minutes in PostGIS. My PostGIS table is approximately 950MB in size, while my CouchBase database, with just the data is 3.7GB, so there is a fairly large difference there.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b><span class="Apple-style-span" style="font-size: large;">Creating the Spatial View</span></b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Once the data is loaded in Couchbase, the next step is to index the spatial features. To do this we create a spatial view. I hit a few snags with this step due to the documentation not being really obvious, and because I was using Futon to create the view, and probably because it was a little new to me generally, but the steps below show how to create a spatial view using Futon.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-kTkyQQbnIhE/Tqbj8d8GkjI/AAAAAAAAAFs/k6lozSJT09s/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="369" src="http://3.bp.blogspot.com/-kTkyQQbnIhE/Tqbj8d8GkjI/AAAAAAAAAFs/k6lozSJT09s/s640/1.png" width="640" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In Futon, click the name of the database in the table list on the overview page. The resulting screen will by default show the first 10 documents in the database.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-0SiaCdcE0mI/Tqbj-K4R0NI/AAAAAAAAAF0/GzMQuU0n-Tk/s1600/2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://4.bp.blogspot.com/-0SiaCdcE0mI/Tqbj-K4R0NI/AAAAAAAAAF0/GzMQuU0n-Tk/s640/2.png" width="640" /></a></div><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In the top-right part of the screen, expand the <b>View </b>drop down and select <b>Temporary view...</b>. This will display the View designer page, showing a pane for defining the Map and Reduce functions.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-Xveey17SUqE/Tqbj_PFML6I/AAAAAAAAAF8/ydm6pFiLJmA/s1600/3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://1.bp.blogspot.com/-Xveey17SUqE/Tqbj_PFML6I/AAAAAAAAAF8/ydm6pFiLJmA/s640/3.png" width="640" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Delete the default function defined in the Map pane, and click the <b>Save As...</b> button. Choose a name for the design document that the view will be saved into, and a name for the view. When the Save button is clicked, an error will be displayed due to the missing Map function. This was done for a reason which I will discuss soon. Regardless of the error, the design document will be created, with the view.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-__9nEdt_mck/TqbkAi-UUaI/AAAAAAAAAGE/XDg8NB5tG6Y/s1600/4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://1.bp.blogspot.com/-__9nEdt_mck/TqbkAi-UUaI/AAAAAAAAAGE/XDg8NB5tG6Y/s640/4.png" width="640" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-tetr1gixPzM/TqbkCBs7pKI/AAAAAAAAAGM/AeWYpVaMKQI/s1600/5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://1.bp.blogspot.com/-tetr1gixPzM/TqbkCBs7pKI/AAAAAAAAAGM/AeWYpVaMKQI/s640/5.png" width="640" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Expand the <b>View </b>drop down again and select <b>Design documents</b>. This will display the list of design documents in the database.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-vTJi4reb2VE/TqbkDZgMsnI/AAAAAAAAAGU/OocKJJ3WKnI/s1600/6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://4.bp.blogspot.com/-vTJi4reb2VE/TqbkDZgMsnI/AAAAAAAAAGU/OocKJJ3WKnI/s640/6.png" width="640" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Choose the document that was just created. The resulting screen will show the structure of the document, including a property called <b>views</b>, which will show a structured item of the name defined in the save screen for the view.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-i04X9JQK2hs/TqbkEq3QC0I/AAAAAAAAAGc/k7RCHL66gzU/s1600/7.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://3.bp.blogspot.com/-i04X9JQK2hs/TqbkEq3QC0I/AAAAAAAAAGc/k7RCHL66gzU/s640/7.png" width="640" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The GeoCouch documentation defines that a spatial view will not be inside a <b>views</b> property, but a property called <b>spatial</b>. So, rename the views property to spatial.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The spatial view will be named the same as the view name that was defined in the save screen, but instead of having a child property called map, which represents the spatial view function, the view name will be the property that represents the function, e.g. the example shows a view called state_1_sp which has a property called map which has a blank string value which is the function that was not saved. This should instead be a property called state_1_sp which has a string value that represents the spatial view.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-w_0T3FVbBGI/TqbkF9xe4CI/AAAAAAAAAGk/g4dd-f62ync/s1600/8.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://2.bp.blogspot.com/-w_0T3FVbBGI/TqbkF9xe4CI/AAAAAAAAAGk/g4dd-f62ync/s640/8.png" width="640" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The spatial view must emit a GeoJSON value as the key, which in the case of my data is the geometry property, because each of my documents is a GeoJSON Feature. In my case I decided to just return the id as the value, because the result from the index will be a list of index keys with a property called geometry, which in essence is the same as a GeoJSON feature, all that I would need to do is create a type property with a string value of Feature for Leaflet to render the data as a FeatureCollection.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
<script src="https://gist.github.com/1313388.js?file=CouchbaseSpatialView.js">
</script><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-bbK7IvPlPrM/TqbkHW4EodI/AAAAAAAAAGs/d_YitSSYjxo/s1600/9.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://3.bp.blogspot.com/-bbK7IvPlPrM/TqbkHW4EodI/AAAAAAAAAGs/d_YitSSYjxo/s640/9.png" width="640" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">When the spatial view has been defined in the document, save the document. Nothing will happen in Futon at this point because it is not aware of spatial views. To start generating the spatial view, open another browser window and supply the following url </span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">http://127.0.0.1:5984/<database>/_design/<design_document>/_spatial/<spatial_view></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This will cause GeoCouch to create the spatial view before trying to return the results. NOTE: the spatial view for my dataset took around 6 hours to create, and took up 85GB in space, which is significantly larger and took longer to create than the index in PostGIS which is about 200MB.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I mentioned above about saving the initial view without a function, which I recommend because I misunderstood the documentation and ended up creating a normal view with a prefix of "spatial" which resulted in Futon creating a typical View with the Map function, which took 35 hours to complete and took up approx 130GB. By removing the Map function before saving, I was able to get around Futon attempting to create the View. </span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Once the Spatial View is created the features within a particular extent can be retrieved using the bbox query string argument appended to the url above - </span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">http://127.0.0.1:5984/<database>/_design/<design_document>/_spatial/<spatial_view>?bbox=<lower_left_long>,<lower_left_lat>,</span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><upper_right_long>,<upper_right_lat></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Calling CouchBase from Leaflet Application</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">My solution for this example is almost identical to <a href="http://boomphisto.blogspot.com/2011/07/nodejs-express-leaflet-postgis-awesome.html">the second post I did on GeoJSON</a> where Leaflet is used to render GeoJSON retrieved from the server as the map is panned. The difference in this case is that I am calling the REST interface for GeoCouch to retrieve the required spatial data as GeoJSON using the url defined above.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
<script src="https://gist.github.com/1313195.js?file=LeafletCouch.html">
</script><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As you can see from the snippet of the GeoCouch bbox query result below, the spatial view results almost match the structure of a FeatureCollection as defined in the <a href="http://geojson.org/geojson-spec.html">GeoJSON specification</a>, except that there are a few other properties on the root object and the row item objects in the GeoCouch results. The rows property of the root object can be seen as synonymous with the features property of a FeatureCollection. Each item in the rows array needs to have a "type" property defined as "Feature", and then the geometry property is exactly the same as the geometry property of a Feature. By making these simple alterations to the result, the structure can then be passed to Leaflet to render the shapes.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
<script src="https://gist.github.com/1316083.js?file=GeoCouchSpatialViewResult.js">
</script><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The problem I did strike with this application is that the Couchbase instance is considered a different domain to my web server hosting the html Leaflet based application page. Because of this I had problems making calls to Couchbase where the request would succeed, but would pass back no data. This was due to cross domain scripting restrictions in the browser.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">To resolve this I switched the jQuery ajax call to use JSONP, but I still had problems with the request. It took a little digging, but I found that Couchbase does not allow JSONP requests by default, so this needs to be switched on by clicking on <b>Configuration</b>, and then scrolling to the <b>httpd </b>section, and then change <b>allow_jsonp</b> to<b> true</b>.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-V38TRw4RWXg/TqbkI5eVccI/AAAAAAAAAG0/fGJbIhhVKX0/s1600/10.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://3.bp.blogspot.com/-V38TRw4RWXg/TqbkI5eVccI/AAAAAAAAAG0/fGJbIhhVKX0/s640/10.png" width="640" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Once that change was saved, the application worked as desired, although the performance from Couchbase is a lot slower compared with the previous solution using ExpressJS/NodeJS and PostGIS, taking around 8 seconds in Couchbase to perform operations that would take 0.5 of a second in PostGIS.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Conclusion</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b><br />
</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Looking back at the evolution of my investigations into GeoJSON it has been interesting to see the depth of interoperability I was able to achieve. When I first began looking I had no idea that I would be able to store my geometry, as an object in a database, and return it as data straight to the browser for rendering.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">While the size/time to create spatial indexes in CouchDB is much larger/longer than PostGIS, I think it is a platform that will improve over time.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">One of the things to remember is that my dataset is a large contiguous layer of data that spans a very large area and possibly isn't really suited for high performing visual rendering of dynamically retrieved data.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">CouchDB has other benefits such as the distributed architecture that allows it to scale out, as well as Couchbase having a mobile solution as well, which when combined with the master-master replication scheme could enable some compelling mobile solutions.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">It would be interesting to investigate Couchbase using a membase server spanning multiple machines, and see if the spatial indexing improves with the parallel index lookup.</span>stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com4tag:blogger.com,1999:blog-392007053337460683.post-72588270721570032922011-08-08T15:47:00.000-07:002011-08-08T15:47:28.943-07:00Nought to PostGIS in 6.3 seconds…almost<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In a<a href="http://boomphisto.blogspot.com/2011/07/nodejs-express-leaflet-postgis-awesome.html"> recent post</a> I decided to investigate some technologies that were new to me, such as <a href="http://leaflet.cloudmade.com/">leaflet</a>, <a href="http://nodejs.org/">node.js</a> and <a href="http://expressjs.com/">express</a>, but in addition to these I had never interacted with <a href="http://www.postgresql.org/">PostgreSQL</a>/<a href="http://postgis.refractions.net/">PostGIS</a> before either, so setting up PostGIS was part of my learning exercise. Being a noob when it came to PostgreSQL and PostGIS, and fairly inexperienced in linux generally, I was able to find the information I needed to get going, but I thought I would pool it together in one location so others can see what I had to do to get my spatial database and queries working in PostGIS, for any other new comers to PostGIS to learn from as well.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<a name='more'></a><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Linux</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I did my setup on a VirtualBox installation of Ubuntu 11.04. My aim was to spend as little time as possible mucking around with installation of software, and maximise time spent on my investigations of leaflet and node.js, so where possible I wanted to install via the published packages available in my installation of Ubuntu without having to download any source code and compile my own versions of libraries. One of my other reasons for wanting to install known published versions was that I intended to use node.js packages to access the PostGIS database, so I wanted to know up front what the documented version compatibilities were, and be able to use the known version as a starting point for investigating why my code might not work, but in the end I didn't have any problems, which is cool :)</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Install PostgreSQL</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">PostgreSQL was relatively easy to install using the packages in Ubuntu, which I installed from the command line, but you could also search for and install these packages in the Synaptic Package Manager.</span><br />
<pre class="brush: shell;">sudo apt-get install postgresql postgresql-client postgresql-contrib pgadmin3
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Install PostGIS</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">PostGIS is an additional install on top of PostgreSQL. In Ubuntu 11.04 the version of PostgreSQL installed is 8.4, so the package required to install PostGIS is postgresql-8.4-postgis, which can be installed with the following command.</span><br />
<pre class="brush: shell;">sudo apt-get install postgresql-8.4-postgis</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Change the admin password</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">When the system is installed, a good practice is to set the administrative user's password. This is done by setting the password for the postgres user in the database, but in addition to this we change the linux user account password as well, so that they are synchronised. </span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Database Login</b></span><br />
<pre class="brush: shell;">sudo su postgres -c psql postgres
postgres=# ALTER USER postgres WITH PASSWORD '<password>';
postgres=# \q
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Ubuntu Login</b></span><br />
<pre class="brush: shell;">sudo passwd -d postgres
sudo su postgres -c passwd
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Now enter the same password that you used for the postgres database login above.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Remote Access</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">For my purposes I did not need to access the database from a remote machine because I indended to run Node.js on the same machine, but if you need to configure remote access for the database, have a look at the following <a href="http://www.stuartellis.eu/articles/postgresql-setup/#enabling-remote-use">page</a> by Stuart Ellis for further information.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Create a template database</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">To make it easier to create new spatial databases in future, the steps below go through a procedure to create a template database from which new databases can be spawned.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Create the template database and run the PostGIS scripts to add the required database objects for PostGIS</span><br />
<pre class="brush: shell;">sudo su postgres
createdb postgistemplate
createlang plpgsql postgistemplate
psql -d postgistemplate -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
psql -d postgistemplate -f /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql
psql -d postgistemplate -f /usr/share/postgresql/8.4/contrib/postgis_comments.sql
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">...and then test it</span><br />
<pre class="brush: shell;">psql -d postgistemplate -c "SELECT postgis_full_version();"</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Create Roles/Users</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The template database will be configured with a role called <i>SpatialGroup</i>. Users will be assigned as members of the <i>SpatialGroup </i>role if they need to read or update spatial data. The database can be opened up/locked down accordingly to give those users the bare permissions they need to the spatial data.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">For now I am only using a single login called <i>spatial</i>, which will also be the owner of the PostGIS tables. This is the user I connected to the database with in my previous example <a href="http://boomphisto.blogspot.com/2011/07/nodejs-express-leaflet-postgis-awesome.html">post</a>.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The following commands create the role and user described above. Make sure to choose a password for the login and replace the placeholder in the sql below.</span><br />
<pre class="brush: shell;">psql -d postgres
postgres=# CREATE ROLE SpatialGroup NOSUPERUSER NOINHERIT CREATEDB NOCREATEROLE;
postgres=# CREATE ROLE spatial LOGIN PASSWORD '<password>' NOINHERIT;
postgres=# GRANT SpatialGroup TO spatial;
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Assign permissions to template database</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">We need to assign permissions for the postgistemplate tables (geometry_columns and spatial_ref_sys) which will be owned by the spatial user.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Exit from the previous connection (type \q), and connect to the postgistemplate database as the postgres user</span><br />
<pre class="brush: shell;">psql -d postgistemplate</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Then assign the permissions...</span><br />
<pre class="brush: shell;">postgistemplate=# ALTER TABLE geometry_columns OWNER TO spatial;
postgistemplate=# ALTER TABLE spatial_ref_sys OWNER TO spatial;
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Create a schema for your spatial data (its best to store the GIS related tables in a custom schema rather than the public schema to aid future maintenance or administration of security)</span><br />
<pre class="brush: shell;">postgistemplate=# CREATE SCHEMA SpatialSchema AUTHORIZATION spatial;
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
Exit from the connection (type \q)<br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Create a database from the template</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Now we can create new spatial databases from the template we created</span><br />
<pre class="brush: shell;">createdb -T postgistemplate -O spatial SpatialDatabase
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Load Data</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">You can use the shp2pgsql utility to load you spatial data from your shapefile into PostGIS. There is documentation on the command line arguments in the <a href="http://postgis.refractions.net/docs/ch04.html#shp2pgsql_usage">PostGIS documentation</a>. Some of the notable arguments are <i><b>I</b></i>, which will create an index on your table after the data is loaded, and <i><b>G</b></i> which will create your shapes as a geography data type rather than a geometry data type, but you must make sure that your data is already in WGS84 long/lat.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I also used the <i><b>D</b></i> argument which the documentation indicates will run faster for large data sets, such as the one I am working with.</span><br />
<pre class="brush: shell;">shp2pgsql -I -c -G -D /media/queensland/cadastre.shp SpatialSchema.Cadastre | psql -d SpatialDatabase -U spatial
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Indexes</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Earlier in the year I did a <a href="http://boomphisto.blogspot.com/2011/04/black-art-of-spatial-index-tuning-in.html">post </a>on spatial indexes in SQL Server and how to analyse spatial data to improve spatial indexes. It is interesting to highlight that indexes in SQL Server are grid based, which forces us to be mindful of our data distribution when configuring the index grid parameters.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Contrary to this approach, PostGIS uses an index type called GiST or Generalized Search Tree. This index structure is used for indexing data that is too complex to be indexed using typical single dimensioned BTrees or something similar. For GIS data the GiST index stores index topology such as other features contained within a feature, or which features overlap a feature etc.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">PostGIS uses an RTree implementation of the GiST index structure to lookup GIS data.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">If you need to create a spatial index on a table, and didn't set the command line argument in the shp2pgsql utility, then you can easily create a spatial index with the following command</span><br />
<pre class="brush: sql;">CREATE INDEX IX_SP_state_1 ON state_1 USING GIST ( geom )
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">...where IX_SP_state_1 is the desired index name, state_1 is the name of the table, and geom is the name of the geometry/geography column.<br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Is my index being used by my query?</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">A problem I encountered in my example <a href="http://boomphisto.blogspot.com/2011/07/nodejs-express-leaflet-postgis-awesome.html">post</a> was that my spatial query which used the ST_Intersects function in the filter expression would not utilise the spatial index, causing the query shown below to run for more than 20 seconds. I was able to confirm that the query was not using the index by checking the query plan in pgAdmin.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">If you execute the statement with the <i>Explain Query</i> button the will result will show the query plan, which should show whether an index will be employed on not.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In the case of the following statement</span><br />
<pre class="brush: sql;">select ST_AsGeoJSON(geog)
from SpatialSchema.Cadastre
where ST_Intersects(geog, ST_GeogFromText('SRID=4326;POLYGON((152.753145 -27.611423,152.753885 -27.613172,152.757018 -27.61244,152.755065 -27.610358,152.753145 -27.611423))'));
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I get the query plan</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-BRtsS_YLDIs/Tj_kFNT-ylI/AAAAAAAAAD0/ICCIEhCYDBA/s1600/QueryPlanTableScan.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="293" src="http://2.bp.blogspot.com/-BRtsS_YLDIs/Tj_kFNT-ylI/AAAAAAAAAD0/ICCIEhCYDBA/s400/QueryPlanTableScan.jpg" width="400" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<br />
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">"Seq Scan on state_1 (cost=0.00..789592.03 rows=860124 width=1082)"</span></div><div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">" Filter: st_intersects(geog, '0103000020E61000000100000005000000AB048BC319186340AF05BD37869C3BC03D7E6FD31F186340AAD216D7F89C3BC0770FD07D3918634022C32ADEC89C3BC0562B137E291863401BDA006C409C3BC0AB048BC319186340AF05BD37869C3BC0'::geography)"</span></div><div lang="en-US" style="font-family: Calibri; font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><br />
</div><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This indicates my query will perform a table scan to select the matching records. As discussed in my earlier <a href="http://boomphisto.blogspot.com/2011/04/black-art-of-spatial-index-tuning-in.html">post</a>, this data set represents all the cadastral boundaries in Queensland, so it is a significant table to be performing a table scan over.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I was able to improve the query by including the following expression to my filter before my ST_Intersects() function.</span><br />
<pre class="brush: sql;">geog && ST_GeogFromText('SRID=4326;POLYGON((152.753145 -27.611423,152.753885 -27.613172,152.757018 -27.61244,152.755065 -27.610358,152.753145 -27.611423))')</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The && operator instructs the query engine to select the features whose bounding boxes intersect the shape's bounding box. </span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This results in a smaller sample that can then be filtered with the ST_Interesects() filter.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The query plan for this query is shown below, indicating that it will use the index to perform an index scan.</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-r0dSMUVXXXo/Tj_kSQWnF2I/AAAAAAAAAD4/FgZXjIpcE14/s1600/QueryPlanIndexScan.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="295" src="http://4.bp.blogspot.com/-r0dSMUVXXXo/Tj_kSQWnF2I/AAAAAAAAAD4/FgZXjIpcE14/s400/QueryPlanIndexScan.jpg" width="400" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<br />
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">"Index Scan using ix_sp_state_1 on state_1 (cost=0.00..9.61 rows=1 width=1082)"</span></div><div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">" Index Cond: (geog && '0103000020E61000000100000005000000AB048BC319186340AF05BD37869C3BC03D7E6FD31F186340AAD216D7F89C3BC0770FD07D3918634022C32ADEC89C3BC0562B137E291863401BDA006C409C3BC0AB048BC319186340AF05BD37869C3BC0'::geography)"</span></div><div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">" Filter: st_intersects(geog, '0103000020E61000000100000005000000AB048BC319186340AF05BD37869C3BC03D7E6FD31F186340AAD216D7F89C3BC0770FD07D3918634022C32ADEC89C3BC0562B137E291863401BDA006C409C3BC0AB048BC319186340AF05BD37869C3BC0'::geography)"</span></div><br />
<br />
stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com2tag:blogger.com,1999:blog-392007053337460683.post-5589280443786672482011-07-26T06:41:00.000-07:002014-02-02T03:29:37.578-08:00Node.js + Express + Leaflet + PostGIS = Awesome Maps<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Recently I <a href="http://boomphisto.blogspot.com/2011/07/simple-maps-using-geojson-and-tile5.html">blogged</a> about using <a href="http://geojson.org/">GeoJSON</a> to create a very lean application for serving dynamic maps. To date, all of my posts have related in some way to Microsoft technologies because that is my environment, day-in, day-out, but I thought it would be interesting to create a similar solution to the last blog post using technologies I haven't used before - why? - because it's good to learn something new, and these libraries are really cool!</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
</div>
<a name='more'></a><br />
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-size: large;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>The Objective</b></span></span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As you would recall from last time, the objective was to create a map with an <a href="http://www.openstreetmap.org/">Open Street Map (OSM)</a> base layer, which overlays the cadastral boundaries as a layer that is retrieved when the user pans or zooms around the map.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The previous solution achieved this using <a href="http://www.tile5.org/">Tile5</a> to render the map on the browser, and make ajax requests to an ASP.Net MVC application which would retrieve the cadastral boundaries for the current view extent from SQL Server, and return them as GeoJSON structured objects to Tile5 to render on the map.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The new solution uses a mapping library called <a href="http://leaflet.cloudmade.com/">Leaflet</a>, which makes requests to a <a href="http://nodejs.org/">Node.js</a> application written using the <a href="http://expressjs.com/">Express</a> framework, which makes requests to a <a href="http://www.postgresql.org/">PostgreSQL</a>/<a href="http://postgis.refractions.net/">PostGIS</a> database server to supply the requested spatial data, so the solution is using Javascript from end to end .</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Leaflet</b></span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><a href="http://leaflet.cloudmade.com/">Leaflet </a>is a lot like <a href="http://www.tile5.org/">Tile5 </a>- it is a javascript mapping library released by <a href="http://cloudmade.com/">CloudMade</a> and exposes functionality to define maps in a browser. The library allows developers to include <a href="http://www.openstreetmap.org/">OSM</a> data layers via CloudMade services, and has facilities to render GeoJSON, just like Tile5 did in the previous post.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Express</b></span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><a href="http://expressjs.com/">Express </a>is a javascript web development framework built on <a href="http://nodejs.org/">Node </a>which gives you the ability to write web sites in a similar way to ASP.Net MVC. The interesting thing about this solution is that it uses javascript both on the client (in the browser) and the server (hosted by Node)</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">View templates in Express can be written in <a href="http://jade-lang.com/">Jade </a>or <a href="http://en.wikipedia.org/wiki/Haml">Haml</a>, of which I chose the latter because I am using <a href="http://www.jetbrains.com/webstorm/">JetBrains' WebStorm</a> as an IDE, and it had native support for Haml syntax checking. Being a noob I thought that would help :)</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>PostgreSQL</b></span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The reason I chose <a href="http://www.postgresql.org/">PostgreSQL </a>is that it appears to have a very rich spatial implementation with the <a href="http://postgis.refractions.net/">PostGIS </a>module, seeing as there were javascript libraries to perform database access I decided it would suite my purposes.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The first step in the development process was install PostgreSQL, and PostGIS and import my Queensland cadastral data into it. The spatial indexing in PostGIS is a little different from SQL Server, and is worthy of a future blog post, suffice to say there are some idiosyncrasies in my spatial queries below that warrant a second glance.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Lets check out the code…</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Installing Express</b></span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Express can be installed by installing Node, and <a href="http://npmjs.org/">NPM (Node Package Manager)</a>, which simplifies the installation of Node based packages. Once these are installed you can install Express. </span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I found that there were some dependencies for Node and NPM that I didn't have on my fresh installation, so I had to install the following -</span></div>
<pre class="brush: bash;">sudo apt-get update
sudo apt-get install git-core curl build-essential openssl libssl-dev</pre>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I also encountered a problem installing Express because I cloned the latest version of Node from GitHub, and it wasn't compatible with the later version, so I had to download version 0.4.10 which I extracted and then ran the following commands to install -</span></div>
<pre class="brush: bash;">./configure
make
sudo make install
node -v</pre>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The final command should show the version of Node that is installed.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">To install NPM -</span></div>
<pre class="brush: bash;">curl "http://npmjs.org/install.sh" | sudo sh
npm -v
</pre>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The final command should show the version of NPM that is installed.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I then installed Express -</span></div>
<pre class="brush: bash;">npm install -g express</pre>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Express can then be used to spawn a template application, and install its dependencies using the following commands -</span></div>
<pre class="brush: bash;">express /home/tjackson/geojsonexample && cd /home/tjackson/geojsonexample
npm install -d
</pre>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This will create an Express application with a basic structure - </span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-zUs5qTRm87M/Ti629bWM6dI/AAAAAAAAADs/Dqy1y2PnHGg/s1600/ExpressStructure.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-zUs5qTRm87M/Ti629bWM6dI/AAAAAAAAADs/Dqy1y2PnHGg/s1600/ExpressStructure.jpg" /></a></div>
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br />
<ul>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">node_modules - contains all the framework libraries</span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">public - contains all the folders that will be exposed on the web server, i.e. images, javascripts, stylesheets</span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">views - contains all the view templates</span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">app.js - is the main application script that is executed in Node, and contains all the route logic, and possibly your controllers</span></li>
</ul>
</div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Controllers</b></span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Express doesn't come with the concept of Controllers straight out of the box, but there is an MVC example in the <a href="https://github.com/visionmedia/express/tree/master/examples">GitHub source branch</a> that shows how to achieve a structure with Controllers in separate javascript files which are imported when the server starts up. The routes must then be configured to point at the required controllers.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This example is so basic, with only two server actions, that the business logic has been included in the app.js file.</span></div>
<script src="https://gist.github.com/stoolrossa/8766787.js"></script>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The changes I made to the originally generated app.js file were</span><br />
<ul>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: 15px;">importing pg, which is the PostgreSQL database access library</span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: 15px;">setting the view engine to haml, but this didn't execute my haml files correctly, so I installed haml-js (npm install hamljs)</span></li>
</ul>
</div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />You can see that the RetrieveCadastre action passes the body through to the function, which represents the bounds of the map to retrieve cadastral information from the database, which you can see being used to build up the SQL statement.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The interesting part of the SQL statement is that it looks like it is using the same filter twice in the where expression. In fact the first part is using the && operator which instructs the query engine to use the spatial index to do an index seek using the bounding box of the filter to find features whose bounding boxes also intersect the filter. This will result in more matches than we expect because some feature's bounding rectangles will intersect our area without the shape inside actually intersecting the area, so what we do is then filter the index results such that the underlying geometries are intersecting our spatial area. For my specific purposes the first filter would be enough, but I included the example for posterity.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The other interesting thing about the SQL is that the selection is returning the GeoJSON of the resulting objects using the <i>ST_AsGeoJSON</i> function. This is really cool because it saves me from having to parse the result into a GeoJSON structure to send back to the client.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">After the statement is executed, you can see that it forms a javascript object for each result feature, using <i>JSON.parse</i>, and then adds this to a feature collection object, which is structured according to the GeoJSON standard. This collection is then returned to the client for rendering. Wow, that’s even leaner than last time round!</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Views</b></span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The only view I have is for displaying the initial map. As you can see it is much like the Tile5 example from last time around, but with some slight interface changes for Leaflet's differences.</span></div>
<script src="https://gist.github.com/stoolrossa/8766821.js"></script>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The obvious main difference is the use of Haml rather than Razor for the view template itself. The one interesting experience I had with Haml was problems with the indenting, which took me back to the days of Fortran77. The thing I had to watch was the linespace between my javascript functions, and that there was the same amount of indenting on the blank lines (spaces from left) as the number of spaces to the start of the function statements. I guess this is so that the resulting html is indented according to the developer's intentions, considering closing tags are not required in the Haml markup.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Includes</b></span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The final thing to do was include my client side dependencies in the public folder, so my Express application looked like the structure blelow-</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-XUp5Hpxr2n0/Ti69hJAJ9zI/AAAAAAAAADw/c06ekvGcBnQ/s1600/ExpressStructureComplete.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-XUp5Hpxr2n0/Ti69hJAJ9zI/AAAAAAAAADw/c06ekvGcBnQ/s1600/ExpressStructureComplete.jpg" /></a></div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Conclusion</b></span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As you can see, the code to achieve the solution is even leaner than the previous solution, mainly due to PostGIS being able to return the database results already formatted into GeoJSON.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<br /></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The example application gives a little peek into the sophistication that can be achieved using javascript as both a client and server side solution. It is also surprisingly zippy considering that I have the entire set of Queensland cadastral boundaries in the PostgresSQL database on a low spec VirtualBox instance of Ubuntu desktop, with the cadastral boundaries refreshing in under a second on maps of the extent of the example below.</span></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-CjXMWqcZges/Ti6xKxVxhOI/AAAAAAAAADo/jIVLYhYP7Dc/s1600/LeafletNodeExpressPostGISMap.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-CjXMWqcZges/Ti6xKxVxhOI/AAAAAAAAADo/jIVLYhYP7Dc/s640/LeafletNodeExpressPostGISMap.jpg" height="540" width="640" /></a></div>
<div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;">
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div>
stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com13tag:blogger.com,1999:blog-392007053337460683.post-13997102653401194572011-07-01T05:38:00.000-07:002011-07-01T05:57:35.905-07:00Simple Maps Using GeoJSON and Tile5<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I've been wanting to post about <a href="http://geojson.org/">GeoJSON</a> for a while now because I like what the standard provides in terms of simplifiying interoperability. The thing is, GeoJSON is just a standard structure for geometry and features represented in <a href="http://www.json.org/">JSON</a>, so there isn't much for me to talk about in regard to GeoJSON itself. So what I decided to do is see how I could utilise GeoJSON to create a very simple mapping application by simplifying communication between tiers, and highlight how JavaScript mapping libraries such as <a href="http://www.tile5.org/">Tile5</a> can help you achieve your vision of simplicity.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<a name='more'></a><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">GeoJSON</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As I said above <a href="http://geojson.org/">GeoJSON</a> is a standard for geometries to be represented in JSON. For the uninitiated <a href="http://www.json.org/">JSON</a> stands for JavaScript Object Notation, and is basically a way of serialising a JavaScript object in a very simple, human readable form. JSON is very similar to the syntax in c# for initialising an anonymous type, for example</span><br />
<br />
<br />
<pre class="brush: javascript;">var geoJsonPolygon = { "type": "Polygon",
"coordinates": [
[ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ]
]
};
</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Once the object is initialised its properties can be accessed as usual, for example</span><br />
<br />
<br />
<pre class="brush: javascript;">var geoJsonType = geoJsonPolygon.type;
</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The GeoJSON standard describes a JSON structure such that a JavaScript object can be created to represent a collection of geometric features, and it is this standard that provides applications with the ability to communicate geometry data in a standard way.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The standard allows for collections of feature objects, of which a feature represents a geometry as well as a list of non-geometric properties. Each geometry object has a property describing its type, i.e. point, line, polygon, etc, as well as a list of coordinates.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">GeoJSON Example - The Objective</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">To illustrate how useful GeoJSON is, I decided to create a simple mapping application utilising an ASP.Net MVC web application, which exposes a <a href="http://www.tile5.org/">Tile5</a> based map in the browser, with a base image layer supplied by one of the <a href="http://www.openstreetmap.org/">Open Street Map (OSM)</a> data providers that Tile5 supports. In addition to this, as the user pans around the map, the Tile5 functionality will call back to the web server to retrieve cadastral boundaries within the extent of the current view of the map, and display the boundaries as an overlay layer. The cadastral boundary data will be supplied as GeoJSON to Tile5 to render on the browser.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The aim is to see how simple and minimal I can make the code to retrieve/marshall/render spatial data between Tile5 and my SQL Server database.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Will the real GeoJSON.Net please stand up?</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Obviously, the ability to communicate between tiers requires that each tier can construct and consume JSON syntax. On a JavaScript client this is fairly simple because JSON relates intimately to the JavaScript, but what do we do on our web server running ASP.Net?</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The objective is that we want to interact with an object model both on the client and the server, and just use JSON to communicate between each tier, so we need to have an object model that represents the same structure as GeoJSON that we can use in C#, and then be able to serialise this object model to JSON.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">At first I thought I might be able to reuse an open source .Net library to model GeoJSON and take care of JSON serialisation, but a search of GeoJSON.Net yeilds a number of results (<a href="http://geojsondotnet.codeplex.com/">CodePlex</a>, <a href="http://code.google.com/p/geojson-net/">Google Code</a>, <a href="https://github.com/jbattermann/GeoJSON.Net">GitHub</a>, Assembla), that don't all seem to be related.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In addtion to needing a library to represent GeoJSON, I also wanted to </span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">use the <a href="http://code.google.com/p/nettopologysuite/">Net Topology Suite</a> to work with my geometry results from SQL Server for data access, so after reading through <a href="http://viswaug.wordpress.com/2008/05/22/customizing-the-response-serialization-in-wcf-rest-services">this post</a> by Vish Uma, in which there is a discussion of some of the drawbacks of exposing third party library objects to JSON serialisation due to not having the ability to attribute the required classes/properties, I decided to create my own very simple classes to model GeoJSON that are a little tighter bound to Net Topology Suite geometry.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Data Access Utilising Net Topology Suite</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As I discussed above, I wanted to use the <a href="http://code.google.com/p/nettopologysuite/">Net Topology Suite</a> to interact with the geometries I select from SQL Server. The reason for this is that there are no objects in the .Net Framework (other than with SQL Server specific libraries) that represent the geometry and geography types in SQL Server, so the way I access this data is by retrieving the geometry or geography data in Well Known Binary (WKB) or Well Known Text (WKT) format, and then use the NetTopologySuite.IO classes to read from either of those formats, to return a geometry object.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">It is then a simple step to populate my GeoJSON model and send the results back to the JavaScript client.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">So let's check out the code....</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">MVC</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Controller</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The controller has two simple actions</span><br />
<br />
<ul><li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><i>Index </i></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">- for retrieving the initial map view</span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><i>RetrieveCadastre </i>- for retrieving the cadastral boundaries that are within the requested bounds</span></li>
</ul><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">An interesting thing to point out is that the <i>RetrieveCadastre </i>action takes an argument called <i>bounds</i>, which is an object of type <i>SpatialDataAccess.Bounds</i>. <i>SpatialDataAccess.Bounds</i> is a type I have created as a server side representation of the bounds object that Tile5 uses in the <i>boundsChange </i>event. This illustrates how JSON can be used in both the request and the response in MVC.</span><br />
<br />
<br />
<pre class="brush: csharp;">using System.Web.Mvc;
namespace GeoJSONMvc.Controllers
{
public class GeoJSONMapController : Controller
{
// Display the map view
public ActionResult Index()
{
return View();
}
// Return the requested cadastral information
[HttpPost]
public JsonResult RetrieveCadastre(SpatialDataAccess.Bounds bounds)
{
return this.Json(SpatialDataAccess.Cadastre.Retrieve(bounds));
}
}
}
</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>View</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The view contains the HTML and JavaScript to define the Tile5 map and operations that occur as a map is panned.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The map will be rendered using a canvas element called <i>mapCanvas</i>. When the document loads the map is created, referencing the <i>mapCanvas </i>element, and adding an OSM layer to the map. A function is defined to callback into when the <i>boundsChange </i>event fires, and this in turn calls the <i>requestUpdatedCadastre</i> function, passing the map extents of the map view.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The <i>requestUpdatedCadastre </i>function uses JQuery to execute an ajax call to the <i>RetrieveCadastre </i>MVC action, passing the <i>bounds </i>as part of the request. If the ajax call succeeds we call the <i>parseResponseCadastre </i>function to allow Tile5 to access the GeoJSON features and render them on the canvas.</span><br />
<br />
<br />
<pre class="brush: html;">@{
Layout = null;
}
<!DOCTYPE html>
<html>
<head>
<title>GeoJSONMap</title>
<script type="text/javascript" src="../../Scripts/jquery-1.5.1.min.js"></script>
<script type="text/javascript" src="../../Scripts/tile5.js"></script>
<script type="text/javascript" src="../../Scripts/geo/osm.js"></script>
<script type="text/javascript">
var map;
$(document).ready(function () {
var startPosition = T5.Geo.Position.parse("-27.43247,153.065654");
// initialise the map
map = new T5.Map({
container: 'mapCanvas'
});
map.setLayer('tiles', new T5.ImageLayer('osm.mapquest', {}));
// goto the specified position
map.gotoPosition(startPosition, 17, function () {
map.bind('boundsChange', function (evt, bounds) {
requestUpdatedCadastre(bounds);
});
});
});
function requestUpdatedCadastre(bounds) {
$.ajax(
{
type: "POST",
url: "/GeoJSONMap/RetrieveCadastre",
dataType: 'json',
data: JSON.stringify(bounds),
contentType: 'application/json; charset=utf-8',
success: function (result) {
parseResponseCadastre(result)
},
error: function (req, status, error) {
alert("Unable to get cadastral data");
}
});
}
function parseResponseCadastre(data) {
T5.GeoJSON.parse(
data.features,
function (layers) {
for (var layerId in layers) {
addLayer(layerId, layers[layerId]);
}
}, {
rowPreParse: function (row) {
return row.geometry;
}
});
}
function addLayer(layerId, layer) {
layer.style = 'area.parkland';
map.setLayer(layerId, layer);
}
</script>
</head>
<body>
<div id="mapContainer" style="width: 800px; height: 600px;">
<canvas id="mapCanvas"></canvas>
</div>
</body>
</html>
</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Data Access</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Cadastre</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As you saw above, the <i>RetrieveCadastre </i>looks very simple, calling <i>SpatialDataAccess.Cadastre.Retrieve(bounds)</i>, and this is where it accesses the data, and packages it up in an object model that represents GeoJSON.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The data is retrieved using a spatial query in SQL Server, returning the geography values in WKB, which are parsed using the <i>NetTopologySuite.IO.WKBReader</i> to create a <i>GeoAPI.Geometries.IGeometry</i> based object. This object is used as the source for the classes that I have created to model GeoJSON geometries.</span><br />
<br />
<br />
<pre class="brush: csharp;">namespace GeoJSONMvc.SpatialDataAccess
{
public class Cadastre
{
public static GeoJSON.FeatureCollection Retrieve(Bounds bounds)
{
var featureCollection = new GeoJSON.FeatureCollection();
var connection = new System.Data.SqlClient.SqlConnection("Data Source=localhost;Initial Catalog=Spatial;User ID=spatial;Password=spatial");
string sql = "select geom.STAsBinary() as wkb, id from state_1 with(index(IX_SP_state_1)) where geom.STIntersects(geography::STPolyFromText('POLYGON((" + bounds.min.lon + " " + bounds.min.lat + "," + bounds.max.lon + " " + bounds.min.lat + ',' + bounds.max.lon + " " + bounds.max.lat + ',' + bounds.min.lon + " " + bounds.max.lat + ',' + bounds.min.lon + " " + bounds.min.lat + "))', 4283)) = 1;";
using (connection)
{
connection.Open();
var command = new System.Data.SqlClient.SqlCommand(sql, connection);
using (command)
{
var reader = command.ExecuteReader();
using (reader)
{
while (reader.Read())
{
var wkbReader = new NetTopologySuite.IO.WKBReader();
var sqlGeometry = wkbReader.Read(reader.GetSqlBytes(0).Stream);
var geoJsonGeometry = new GeoJSON.Geometry(sqlGeometry);
geoJsonGeometry.type = GeoJSON.ObjectType.Polygon.ToString();
var geoJsonFeature = new GeoJSON.Feature();
geoJsonFeature.geometry = geoJsonGeometry;
geoJsonFeature.properties.id = reader.GetInt32(1).ToString();
featureCollection.features.Add(geoJsonFeature);
}
}
}
}
return featureCollection;
}
}
}
</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Bounds</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The <i>SpatialDataAccess.Bounds</i> is a fairly simple class that represents two points, <i>min </i>and <i>max</i>, with a <i>lon </i>and <i>lat </i>property representing the coordinates. This type is used as the argument to the <i>RetrieveCadastre </i>action, and is passed through to the <i>Cadastre.Retrieve</i> method to get the spatial features within the bounds.</span><br />
<br />
<br />
<pre class="brush: csharp;">namespace GeoJSONMvc.SpatialDataAccess
{
public class Bounds
{
public Point min { get; set; }
public Point max { get; set; }
}
public class Point
{
public decimal lon { get; set; }
public decimal lat { get; set; }
}
}
</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">GeoJSON Classes</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>GeoJsonObject</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The GeoJSONObject class is a base class that all GeoJSON objects derive from. This class has an implementation for the <i>type</i> property. The <i>type</i> property has been implemented as a string due to .Net serialising enumerations using the integer value rather than the enumeration value name, so to save mucking around I simply supply the object type name as a string. I have an enumeration that lists these types and simply pass the required setting as <i>Enumeration.ToString()</i>.</span><br />
<br />
<br />
<pre class="brush: csharp;">namespace GeoJSONMvc.GeoJSON
{
public abstract class GeoJsonObject
{
public string type { get; set; }
}
public enum ObjectType
{
Feature,
FeatureCollection,
Point,
MultiPoint,
LineString,
MultiLineString,
Polygon,
MultiPolygon,
GeometryCollection
}
}
</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>FeatureCollection</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The FeatureCollection class represents the collection of features. </span><br />
<br />
<br />
<pre class="brush: csharp;">using System.Collections.Generic;
namespace GeoJSONMvc.GeoJSON
{
public class FeatureCollection : GeoJsonObject
{
public FeatureCollection()
{
this.features = new List<Feature>();
}
public List<Feature> features { get; set; }
}
}
</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Feature</b><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
The feature represents a geometry object with properties, i.e. a spatial record with non-spatial attributes.<br />
</span><br />
<br />
<br />
<pre class="brush: csharp;">namespace GeoJSONMvc.GeoJSON
{
public class Feature : GeoJsonObject
{
public Feature()
{
this.properties = new Properties();
base.type = ObjectType.Feature.ToString();
}
public Geometry geometry { get; set; }
public Properties properties { get; set; }
}
}
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Geometry</b><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
A geometry represents a shape, i.e. Point, MultiPoint, LineString, MultiLineString, Polgyon, MultiPolygon, GeometryCollection. The coordinates array represents a 1:n array of coordinates.<br />
</span><br />
<br />
<br />
<pre class="brush: csharp;">namespace GeoJSONMvc.GeoJSON
{
public class Geometry : GeoJsonObject
{
public Geometry()
{
this.coordinates = new decimal[0][];
}
public Geometry(GeoAPI.Geometries.IGeometry shape)
{
this.coordinates = new decimal[shape.Coordinates.Length][];
for(var i = 0; i < shape.Coordinates.Length; i++)
{
var coordinate = shape.Coordinates[i];
this.coordinates[i] = new decimal[2] { System.Convert.ToDecimal(coordinate.X), System.Convert.ToDecimal(coordinate.Y) };
}
}
public decimal[][] coordinates { get; set; }
}
}
</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Properties</b><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
The Properties class simply represents a container type for all properties of a Feature. In the case of the data I am using, there is only one property I am interested in including which is feature id.<br />
</span><br />
<br />
<br />
<pre class="brush: csharp;">namespace GeoJSONMvc.GeoJSON
{
public class Properties
{
public string id { get; set; }
}
}</pre><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-size: large;">Results</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
The image below is a screen shot of the application, which displays the retrieved cadastral boundaries overlaid on the OSM layer. The road names as displayed in OSM are visible in the screen between the blocks of cadastral boundaries. Each time the user pans or zooms, the application retrieves the cadastral boundaries within the map extent and renders them on the map.<br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
While this is a fairly simplistic map application, it shows how easy it is acquire the data in GeoJSON format and render on the map. I put this example together in just a few hours, which was fairly good considering I had not used Tile5 before. The data access logic was very easy to put together.<br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-Sa2s0dlBUu0/Tg20y3gitHI/AAAAAAAAACE/xe882Yi34vs/s1600/MapResult2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="476" src="http://3.bp.blogspot.com/-Sa2s0dlBUu0/Tg20y3gitHI/AAAAAAAAACE/xe882Yi34vs/s640/MapResult2.jpg" width="560" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-size: large;">Conclusion</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
I like that the spatial community was very quick to identify that the emergent use of JSON would benefit from having standards around the structure of spatial data. JavaScript is now ubiquitous in the world of web mapping, so standards in this area provide developers with opportunities to leverage functionality that interacts and adheres to those standards.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">It is also interesting to note that <a href="http://blogs.msdn.com/b/youssefm/archive/2009/07/10/comparing-the-performance-of-net-serializers.aspx">this comparison</a> of serialisation methods in .Net shows JSON serialisation to be one of the better performing methods, resulting in one of the smallest packet sizes.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The reality of this experiment is that all the real work is being done by the Tile5 libraries, which is really cool. All I had to do is provide some simple data access, which was very simple to achieve.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
One of the things to note about the example is that there are no constraints around the amount of GeoJSON data being retrieved and rendered on the client. In the real world there would be more logic around scale dependent layers to limit the volume of data being passed around for larger view extents.stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com4tag:blogger.com,1999:blog-392007053337460683.post-80780888139088008852011-06-03T03:24:00.001-07:002011-06-03T04:03:42.891-07:00Persisted Spatial Calculation Columns<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This blog post discusses how I computed the area values of the features in my spatial tables to improve query/reporting performance during my </span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><a href="http://boomphisto.blogspot.com/2011/04/black-art-of-spatial-index-tuning-in.html">investigation of SQL Server spatial indexes</a>.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<a name='more'></a><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As part of my <a href="http://boomphisto.blogspot.com/2011/04/black-art-of-spatial-index-tuning-in.html">recent spatial index investigation</a>, I performed some analysis on a number of data sets to get a feeling for their spatial density to improve decision making about settings for my spatial indexes.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This analysis involved investigation of relative area of features compared to the entire data set, and data extents, which required summarisation of area values of varied selections of features.</span></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Relatively speaking, the act of performing a calculation of the area of a polygon feature using the STArea method is computationally expensive, so aggregating the values returned from STArea across all data in a table, or performing a selection using the STArea function as a criteria in a filter (e.g. select * from cadastre where geom.STArea() < 2000) can impact performance dramatically. Considering that some of my data sets contained millions of rows of data, this wasn't really desirable.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Persisted Computed Columns</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">One of the features introduced in SQL Server 2005 was Persisted Computed Columns. These columns allow you to define a computed column, but rather than calculating the value of the column when queries are executed, the computed column value is saved in the database. The computed column value is updated when the columns that the value is derived from are updated.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The computed column will then be much faster to perform reporting and queries over.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Here is an example of a table that uses a persisted computed column to store the area (area column) of each polygon in the table, and updates whenever the geometry (geom column) is updated.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<pre class="brush: sql;">create table cadastre(
geom geometry not null,
area as (geom.STArea()) persisted not null
)
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span>stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com0tag:blogger.com,1999:blog-392007053337460683.post-43861162087826298972011-06-01T04:07:00.001-07:002011-06-01T05:25:42.592-07:00SQL Server Invalid Geometry<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">While <a href="http://boomphisto.blogspot.com/2011/04/black-art-of-spatial-index-tuning-in.html">recently investigating SQL Server spatial indexes</a> I encountered an issue regarding invalid geometry in some of the features in my source shape files, so I thought I would do a quick blog post on what this means, and how to detect and resolve these issues in SQL Server.</span><br />
<br />
<a name='more'></a><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">What is valid geometry?</span><br />
<div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">If you research the <a href="http://msdn.microsoft.com/en-us/library/bb933890.aspx">STIsValid</a> method in the SQL Server documentation you will come across an explanation of the method that indicates that it will return whether the geometry instance is <i>"well formed based on its Open Geospatial Consortium (OGC) type"</i>, but what does this mean?</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Well, this checks for basic spatial integrity depending on the data type, such as polygons not having inner and outer rings crossing, and multilines not retracing back over the same points.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Part of the documentation from the <a href="http://www.opengeospatial.org/standards/sfa">OGC standards for simple feature access</a> describe validity of polygons being:</span><br />
<ol><li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Polygons are topologically closed</span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The boundary of a Polygon consists of a set of LinearRings that make up its exterior and interior boundaries</span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">No two Rings in the boundary cross, and the Rings in the boundary of a Polygon may intersect at a Point but only as a tangent</span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">A Polygon may not have cut lines, spikes or punctures </span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The interior of every Polygon is a connected point set</span></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The exterior of a Polygon with 1 or more holes is not connected. Each hole defines a connected component of the exterior</span></li>
</ol><br />
<div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><span class="Apple-style-span" style="font-size: large;">STIsValid Method</span></span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The <a href="http://msdn.microsoft.com/en-us/library/bb933890.aspx">STIsValid </a>method can be used to detect if you have shape features that are invalid. The method will return 0 if the feature is invalid</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">e.g. The following TSQL command will count the number of records in the table with invalid shape features</span></div><pre class="brush: sql;">select count(id) from cadastre where geom.STIsValid() = 0
</pre><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-size: large;">MakeValid Method</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Luckily for us there is a method in SQL Server to fix invalid shape data - <a href="http://msdn.microsoft.com/en-us/library/bb933835.aspx">MakeValid</a>.</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Be aware that MakeValid could alter your shape in a way that you consider incorrect, depending on the original problem with the shape. So you should really investigate any shapes that are invalid using STIsValid to find them, and then if using MakeValid to resolve them, check the resulting shapes produced by MakeValid before performing an update.</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">My sample dataset only had a handful of errors caused by overshoots etc, so I was able to resolve my data with the following command</span></div><pre class="brush: sql;">update cadastre set geom = geom.MakeValid() where geom.STIsValid() = 0
</pre><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">If you have a lot of invalid shapes you may want to perform commits in some kind of unit of work, such as this script that commits every 1000 updated rows</span></div><pre class="brush: sql;">declare @rowCount int
set @rowCount = 9999
while @rowCount > 0
begin
begin tran
update top(1000) cadastre set geom = geom.MakeValid() where geom.STIsValid() = 0
set @rowCount = @@rowcount
commit
end
</pre>stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com2tag:blogger.com,1999:blog-392007053337460683.post-18684934022341254192011-05-19T04:06:00.000-07:002011-05-19T04:06:54.284-07:00Datum Transformations<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In my last post on <a href="http://boomphisto.blogspot.com/2011/05/sql-server-spatial-datums.html">SQL Server Spatial Datums</a> I discussed the SRID setting, and what it relates to in terms of standard mapping datums, but I mentioned that SQL Server is not a GIS and does not have a facility to perform coordinate transformations between datums, so it is important to execute spatial operations on data sets that relate to the same datum.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In this post I will discuss transforming data sets from one datum to another, and focus more specifically on the change of the Australian datum from the Australian Geodetic Datum (AGD) to the Geocentric Datum of Australia (GDA).</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<a name='more'></a></div><div><b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Datum Transformations</span></b></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In my <a href="http://boomphisto.blogspot.com/2011/05/sql-server-spatial-datums.html">previous post</a> I mentioned that SQL Server uses an SRID to define which datum an instance of a geographic data type belongs to, and this allows SQL Server to expose spatial properties such as STLength and STArea. But something that is missing from SQL Server is that it doesn't know the differences between one datum and another, so it cannot transform data from one datum into another. This means that data from one datum cannot be used to perform spatial selections on data from another datum.</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">To facilitate this requirement the data needs to be in a common datum. The easiest way to achieve this is to create spatial layers that use the same datum. If your data layers use different datums, then you will need to transform the data from their current datum into the common datum.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">If you have commercial GIS software such as ESRI's ArcMap, or SafeSoftware's FME products, it is a relatively simple operation to transform from one datum to another because the software knows the parameters required to transform from one datum to another, or one projection to another, but since I was just <a href="http://boomphisto.blogspot.com/2011/04/black-art-of-spatial-index-tuning-in.html">investigating spatial indexes</a> at home, I was more interested in a cheaper option, so I started looking at <a href="http://www.gdal.org/ogr2ogr.html">ogr2ogr</a> which is one of the utilities in <a href="http://fwtools.maptools.org/">FW Tools</a>.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">FW Tools is a set of open source GIS utilities and contains the ogr2ogr utility, which is part of the <a href="http://www.gdal.org/">Geospatial Data Abstraction Library (GDAL)</a>, which is an open source library for manipulating spatial data.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">ogr2ogr is a command line utility that can be used to convert data from one format to another and/or one projection or datum to another.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">For instance, I could use the following command to convert a shapefile of data representing features in grid coordinates in Australian Map Grid (AMG) Zone 56, deriving from the Australian Geodetic Datum (AGD84) data to latitude longitude.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
<pre class="brush: shell;">ogr2ogr -f "ESRI Shapefile" -s_srs EPSG:20356 -t_srs EPSG:4203 CadastreAGD84.shp CadastreAMGZone56AGD84.shp
</pre></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Note that, as discussed in my previous post on <a href="http://boomphisto.blogspot.com/2011/05/sql-server-spatial-datums.html">SQL Server Spatial Datums</a> that the AMG projection is based on the AGD datum, so there is no datum transformation as such, just a conversion from projected coordinates to geographic coordinates.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">If I wanted to convert between datums I would need to provide some transformation parameters. From what I can see ogr2ogr can accept a well known text (WKT) parameter called towgs84, which will convert the datum to the World Geocentric Datum (WGS84) but this requires knowledge of the parameters, and also means that the data can only be transformed to WGS84 using those settings. For my Australian data there is a special datum transformation I can perform that involves a transformation shift based on a distortion file, which I will discuss in further detail below.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I found an interesting <a href="http://www.bostongis.com/?content_name=ogr_cheatsheet">"cheat sheet" post</a> on ogr2ogr on the <a href="http://www.bostongis.com/">Boston GIS site</a>.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">To see more examples of the WKT syntax you can look at the <b>epsg</b> file in the <b>proj_lib</b> directory of the FW Tools installation area. This file will contain the definitions of a lot of the EPSG datums, so there will be examples of defining projections, and ellipsoids, as well as some of the names of ellipsoids recognised by ogr2ogr.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">EPSG Datum Identifiers</span></b><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As you saw above I used the EPSG switch to indicate that I wanted to transform from my projection to my datum. EPSG stands for the European Petroleum Survey Group, and the identifiers represent the same id's that SQL Server uses as the SRID to define the datum in which a geographic data feature is referenced.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">You can find a listing of the EPSG datums on the <a href="http://spatialreference.org/ref/epsg/">spatialreference.org website</a>, but being an Australian resident, some of the EPSG's that I am more interested in are:</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">AGD66 4202</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">AMG Zone 56 20256</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">AGD84 4203</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">AMG Zone 56 20356</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">GDA94 4283</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">MGA Zone 56 28356</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">WGS84 4326</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Datum Transformations in Australia</span></b><br />
<b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></b><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In Australia there was a very important change in datums enacted in 2000. The <a href="http://www.icsm.gov.au/icsm/">Inter-gevernmental Committee on Surveying and Mapping (ICSM)</a> made a decision to change the datum used for standard mapping to use a <a href="http://www.icsm.gov.au/icsm/gda/index.html">geocentric datum</a>. One of the main reasons for this was to provide consistency with coordinates determined from GPS, which has become much more popular since the original datums were defined (i.e. AGD66 in 1966). The explosion in the number of casual map users with smartphones is another group of users that is benefiting from the change in the datum which was unforeseen even at the time the datums were changed.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I guess one of the possible questions is that if this happened 11 years ago, why do we need to care about transforming data from one datum to another? The reason for this is that it is quite conceivable that there are data sets in general population that are still using the older datums (particularly historical data), so we need to be able to convert these data sets to our new datums to be able to overlay them correctly, or perform spatial analysis.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Migrating from AGD to GDA</span></b></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As I mentioned above, 11 years ago the standard datum for Australian mapping changed from the Australian Geodetic Datum (AGD66 or AGD84) to the Geocentric Datum of Australia (GDA94). The AGD66 and AGD84 datums used the same spheroid, but had a slightly different adjustment of the control points in the Australian control network. Other than the parametric differences, such as the semi-major axis and flattening, the noticeable difference between the AGD spheroid and the GDA spheroid is that the origin of AGD is not geocentric with the Earth. The GDA spheroid is the same as the WGS84 spheroid which is used by GPS sattelites, which as I said, means that measurements from GPS are usable on the GDA datum without complex transformation.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">If you are interested in further reading on GDA, check out the <a href="http://www.icsm.gov.au/icsm/gda/gdatm/gdav2.3.pdf">GDA Technical Manual</a> produced by the ICSM</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Just to add a little more confusion to the AGD/GDA acronyms is the change in terminology for projections of AGD from the Australian Map Grid (AMG) to the Map Grid of Australia (MGA) for GDA. Theses standard map grids are both Universal Transverse Mercator (UTM) projections.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The UTM projection is a variation of the Mercator projection. If you use Google Maps you would be familiar with the Mercator projection - it is as though you wrapped a piece of paper around the earth as a cylinder touching the equator. <a href="http://en.wikipedia.org/wiki/Map_projection">Different projections have different characteristics</a>, where a particular geometric aspect is maintained (length, angles, area) at the expense of introducing some type of distortion. You can see in the image below from Google Maps that the Mercator projection shows Greenland to be larger than Australia, but we all know that Greenland is actually smaller than Australia, so it can be seen that the Mercator projection is distorting the data particularly at the poles. In this case the area is being distorted, while angles are maintained.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-IFOx6jnELF0/TcvRE-FTfJI/AAAAAAAAABg/7cO3_S33VLU/s1600/Mercator.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="308" src="http://3.bp.blogspot.com/-IFOx6jnELF0/TcvRE-FTfJI/AAAAAAAAABg/7cO3_S33VLU/s400/Mercator.jpg" width="400" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">On the other hand, you can see that the Mollweide projection below has maintained the relative area of Greenland and Australia, but the angles have not been maintained, showing the data as sheared, or skewed.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-snHlf1yHuTc/TcvRRqZrgxI/AAAAAAAAABk/6iSsbH0HjsI/s1600/Mollweide.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="http://3.bp.blogspot.com/-snHlf1yHuTc/TcvRRqZrgxI/AAAAAAAAABk/6iSsbH0HjsI/s400/Mollweide.jpg" width="400" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Standard mapping projections are defined such that they minimise the distortion while retaining their useful geometric attributes. In the case of UTM, this represents a Mercator projection, except that rather than being defined around the equator, the line touching the surface of the spheroid is vertical, along meridians of longitude, hence the "transverse" terminology. The mapping system is then broken up into small zones to minimise the amount of distortion at the edges.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The AMG and MGA mapping systems are broken into 6 degree zones - see below.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-f7943xHADF0/TcvSLuypW8I/AAAAAAAAABo/USRp9r4kPbk/s1600/AMGZones.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="259" src="http://4.bp.blogspot.com/-f7943xHADF0/TcvSLuypW8I/AAAAAAAAABo/USRp9r4kPbk/s320/AMGZones.jpg" width="320" /></a></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">So the zones that I defined in the ESPG section above relate to the Zones I care most about, living in the South East corner of Queensland.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Transformation of AGD to GDA using a Distortion Grid</span></b><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As I discussed above, I was able to use ogr2ogr to transform projected grid coordinates into the equivalent geographic coordinates of its related datum, but I was not able to transform from AGD to GDA because I didn't have any transformation parameters. If you read through the <a href="http://www.icsm.gov.au/icsm/gda/gdatm/gdav2.3.pdf">GDA Technical Manual</a> you will find that the recommended method (and highest accuracty method) for transforming data is to use a distortion grid to translate the coordinates from one datum to another. Th</span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">e ICSM has <a href="http://www.icsm.gov.au/icsm/gda/gdatm/index.html">released some distortion grid data</a> that can be used to perform this translation.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The cool thing I found was that I could define switches in the ogr2ogr command line utility to use these distortion grids to transform my data.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">My data was transformed from AMG Zone 56 to AGD84, in the example at the top of the post, so I could then use the following command to transform the data to GDA94.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
<pre class="brush: shell;">ogr2ogr -f "ESRI Shapefile" -s_srs "+proj=longlat +ellps=aust_SA +nadgrids=C:\datum\grid84.gsb +units=m +no_defs +wktext" -t_srs EPSG:4283 CadastreGDA94.shp CadastreAGD84.shp
</pre></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">If I wanted to transform directly from grid coordinate data in AMG Zone 56 (on AGD84) then I could use the following command to transform the data to GDA94.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
<pre class="brush: shell;">ogr2ogr -f "ESRI Shapefile" -s_srs "+proj=utm +zone=56 +south +ellps=aust_SA +nadgrids=C:\datum\grid84.gsb +units=m +no_defs +wktext" -t_srs EPSG:4283 CadastreGDA94.shp CadastreAMGZone56AGD84.shp
</pre></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Note the use of the UTM projection (proj=utm), in Zone 56 (zone=56) using the australian spheroid (ellps=aust_SA).</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;">Conclusion</span></b><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As I already mentioned, you never know when you might need to transform your data to another datum in order to overlay it, so you need to know the origin of the data and the datum and/or projection it belongs to. One of the tricky things about projected data is that is just a grid based coordinate system, so quite often it is not clear which projection it belongs to, or even which datum, unless there is adequate metadata describing this.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I found that using the ogr2ogr application was a little difficult to get detailed information about the WKT parameters it uses, but once I got it working I found it to transform the data correctly (I was lucky enough to have some test data I could compare), and was actually very fast to transform.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">One of the important things to ensure is that the data is being transformed as you would expect, especially before running a batch job that transforms lots of layers of data. So make sure you can test the output position of some known points to verify the transformation.</span></div>stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com0tag:blogger.com,1999:blog-392007053337460683.post-53289212962714488872011-05-02T02:33:00.001-07:002011-05-04T06:06:17.896-07:00SQL Server Spatial Datums<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As part of my <a href="http://boomphisto.blogspot.com/2011/04/black-art-of-spatial-index-tuning-in.html">investigation of spatial indexes in SQL Server</a>, I performed some analysis on some spatial data sets, so during the import of my spatial data I had to consider the datum for my data, so I thought I would devote this blog post to a dicussion about datums and spatial reference systems in SQL Server.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<a name='more'></a><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>What is a Datum?</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">What is a <a href="http://en.wikipedia.org/wiki/Datum_(geodesy)">datum</a>? Put simply, a datum is the reference system that a map's coordinates relate to. If we think of a graph for example, the graph's axes represent the datum to which the data in the graph is referenced.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">When we talk about a datum in a mapping coordinate system, we are typically referring to the spheroidal model of the Earth that we are using to represent geographic coordinates (longitude, latitude) against.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>What is a Spheroid?</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In the paragraph above I used the term spheroidal, but what is a <a href="http://en.wikipedia.org/wiki/Spheroid">spheroid</a>? The shape that is the closest approximation for the shape of the Earth is referred to as the spheroid, or ellipsoid. This shape can be thought of as a basketball that is being squished at the north and south poles, such that the line around the equator is the longest distance around the shape.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">So if we know the Earth is spheroidal, then why do I use the plural term datum<b>s</b>? Well, in fact the Earth is not a perfect spheroid. It is actually a shape known as the <a href="http://en.wikipedia.org/wiki/Geoid">Geoid</a>. The Geoid is an undulating three dimensional surface representing the gravitational surface of the earth. I often picture this surface in my mind as a potato with divots and growths, but the undulations are actually fairly subtle.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">A spheroid is a three dimensional ellipse that closely approximates this geoidal surface. But obviously different regions on the earth would have higher or lower average gravitational density, so there are different datums throughout the world that represent spheroids that best approximates the Geoid in their regional area.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>The SQL Server SRID</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In SQL Server, when you create an instance of the geography data type, one of the parameters is an SRID, but what is an SRID? You guessed it - the SRID is an identifier that represents the datum that the geography feature is referenced to. SRID stands for Spatial Reference Identifier.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I couldn't find a lot of information about the SRIDs in SQL Server, but I found that I could perform a query against the spatial_reference_systems system view to see which SRIDs it contained.</span><br />
<pre class="brush: sql;">select * from sys.spatial_reference_systems
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">If you execute this query you will see a column that represents the Well Known Text (WKT) of each SRID, which defines the parameters of the spheroid, but in addition to this has the spheroid name.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Among my data sets I had a shape file of GDA94 (Geocentric Datum of Australia) geographic data representing the cadastral boundaries of the state of Queensland. So I performed the following query to narrow my search.</span><br />
<pre class="brush: sql;">select * from sys.spatial_reference_systems where well_known_text like '%gda%'
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Resulting in SRID 4283 being found. I then used this to import my data using the Shape2SQL utility from <a href="http://www.sharpgis.net/page/SQL-Server-2008-Spatial-Tools.aspx">Morten Nielsen's blog</a>. </span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>What about Projections?</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Ok then, so the next question is - what is a <a href="http://en.wikipedia.org/wiki/Map_projection">projection</a>? A projection is a representation of spheroidal data on a flat piece of paper.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">When we talk about geographic coordinates, the longitude and latitude represent polar coordinates, and differences between points are defined as angular arc measurements on the earth's surface, but these measurements are less meaningful to us than units we interact with every day such as meters and kilometres, or feet and miles. So a projection is a way that we can represent the polar geographic coordinates on a piece of paper in a grid of units we are familiar with.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Like spheroidal datums, different countries usually have standard map grids that are grid representations of their spheroidal datum. In my case I had another set of data in MGA (Map Grid of Australia) coordinates, which is a grid coordinate system based on the GDA94 datum.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Since this data is represented purely as grid values, the data could be represented as geometry shapes using a SRID of 0 when the data was imported.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Comparison Between Datums</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">When creating spatial data it is quite important to define the correct SRID, the first reason being that two spheroids can give slightly different results when calculating distances or areas, simply because of the difference in their mathematical parameters.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">e.g. If I draw a line from the top left corner to the bottom right corner of the extent of my Queensland cadastral data in a GDA94 datum and get the distance of the line, I get a slightly different answer than if I use the AGD66 (Australian Geodetic Datum) datum.</span><br />
<pre class="brush: sql;">declare @line geography
set @line = geography::STLineFromText('LINESTRING(137.9946 -9.142,153.5518 -29.178)', 4283)
print @line.STLength()
set @line = geography::STLineFromText('LINESTRING(137.9946 -9.142,153.5518 -29.178)', 4202)
print @line.STLength()
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Results:<br />
2.74931e+006<br />
2.74932e+006<br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The other important thing about comparisons between datums, is that SQL Server is not a GIS, it is a spatial data storage system, so it won't perform transformations between datums or projections, so you cannot use coordinates from one datum to select features in another datum.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In order to perform spatial selections between data in different datums or projections, a transformation must be done so that the comparison is done in a common datum.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In a future post I will discuss an example where I needed to convert data in an older AMG (Australian Map Grid) projection based on AGD66 to the MGA projection based on GDA94 datum.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><b>Tips</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Just some final thoughts about datums in SQL Server - while the SRID argument allows a geographic instance to define its datum, it serves more of a technical purpose to SQL Server to make spatial calculations such as distances and areas, than actually represent any meta data about the spatial data itself. This is especially true of planar grid based geometry data instances that use a SRID of 0.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I think it is vitally important to record the datum/projection details for different spatial tables in your database, because at some time in the future you will need to know the origin of the data, and possibly how to transform it to another coordinate system.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">For this purpose I think it is worth creating a table within your SQL Server database to define this data for your spatial tables, with the table name, datum, and the projection as an optional column. e.g. local_auth_1, AGD66, AMG Zone 56.</span>stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com0tag:blogger.com,1999:blog-392007053337460683.post-32065446145255698902011-05-01T17:13:00.001-07:002011-05-02T02:39:53.893-07:00Spatial Index Tuning Scripts<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I have posted some of the scripts I used when performing the analysis for my previous post on <a href="http://boomphisto.blogspot.com/2011/04/black-art-of-spatial-index-tuning-in.html">Spatial Index Tuning in SQL Server</a></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<a name='more'></a><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"> <br />
<b>Density Analysis Script</b> <br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: normal;"><br />
This script was used for analysing the maximum spatial density of features in the Geography based data set. This was done to get an indication of the granularity of features compared to the size of the data set in order to make a decision on the granularity of the spatial index that would be created. As discussed in my previous article the index for the Geographic data could not be defined <br />
<br />
The only difference between this script and the Geometry based test script was that the instance of the grid cell polygon was created using <i>geometry::STPolyFromText</i>, supplying 0 as the SRID.<br />
<br />
This script can take a long time to execute on a table with lots of rows, or complicated shapes (or a combination of both), so it is best to apply a spatial index of some sort before executing this script. You will notice the script is using an index hint to get the query engine to use the spatial index.<br />
<br />
</span><br />
<pre class="brush: sql; highlight: [55]">declare @xmin decimal(19,6)
declare @ymin decimal(19,6)
declare @xmax decimal(19,6)
declare @ymax decimal(19,6)
-- Define the data extent
set @xmin = 137.9946
set @ymin = -29.178
set @xmax = 153.5518
set @ymax = -9.142
declare @iterations int
declare @iterationRow int
declare @iterationColumn int
declare @cellWidth decimal(19,6)
declare @cellHeight decimal(19,6)
declare @count int
declare @maxCount int
declare @cellXmin decimal(19,6)
declare @cellYmin decimal(19,6)
declare @cellXmax decimal(19,6)
declare @cellYmax decimal(19,6)
-- The number of rows/columns in the grid of cells that will be analysed
set @iterations = 20
set @iterationRow = 0
set @iterationColumn = 0
set @count = 0
set @maxCount = 0
-- Calculate the size of each grid cell (note that these will be rectangular if the data extent is rectangular)
set @cellWidth = abs((@xmax - @xmin) / @iterations)
set @cellHeight = abs(@ymax - @ymin) / @iterations
while @iterationRow < @iterations
begin
-- Determine the grid cell's y coords from the row we are analysing
set @cellYmin = (@iterationRow * @cellHeight) + @ymin
set @cellYmax = ((@iterationRow + 1) * @cellHeight) + @ymin
set @iterationColumn = 0
while @iterationColumn < @iterations
begin
-- Determine the grid cell's x coords from the column we are analysing
set @cellXmin = (@iterationColumn * @cellWidth) + @xmin
set @cellXmax = ((@iterationColumn + 1) * @cellWidth) + @xmin
-- Select the number of features in the grid cell
select @count = count(id) from state_1 with(index(IX_SP_state_1)) where geom.STIntersects(geography::STPolyFromText('POLYGON((' + CONVERT(varchar(20), @cellXmin) + ' ' + CONVERT(varchar(20), @cellYmax) + ',' + CONVERT(varchar(20), @cellXmin) + ' ' + CONVERT(varchar(20), @cellYmin) + ',' + CONVERT(varchar(20), @cellXmax) + ' ' + CONVERT(varchar(20), @cellYmin) + ',' + CONVERT(varchar(20), @cellXmax) + ' ' + CONVERT(varchar(20), @cellYmax) + ',' + CONVERT(varchar(20), @cellXmin) + ' ' + CONVERT(varchar(20), @cellYmax) + '))', 4283)) = 1;
-- If this is the new maximum count, retain it
if @count > @maxCount set @maxCount = @count
set @iterationColumn = @iterationColumn + 1
end
set @iterationRow = @iterationRow + 1
end
-- Display the maximum number of features found in any grid cell, and relate their relative density/size to the overall extent
print 'Maximum cell records: ' + convert(varchar(10), @maxCount)
print 'Feature size relative to data extent: ' + convert(varchar(20), @maxCount * @iterations * @iterations)
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><br />
<b>Spatial Selection Test Script </b><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: normal;"><br />
This script was used to test the time taken to execute four separate spatial selections on the Geography based data set. The four selections are executed 100 times and the time then averaged to arrive at a result. <br />
<br />
The difference between this script and the Geometry based test script was that the test extents were defined as Geometry variables, and the instance of the Geometry was created using <i>geometry::STPolyFromText</i>, supplying 0 as the SRID.<br />
<br />
</span><br />
<pre class="brush: sql; highlight: [1,2,3,4,7,8,9,10];" rows="20">declare @test1Extent geography
declare @test2Extent geography
declare @test3Extent geography
declare @test4Extent geography
-- Define the selection extents
set @test1Extent = geography::STPolyFromText('POLYGON((153.035517 -27.417948,153.032598 -27.42869,153.049808 -27.431585,153.05161 -27.420271,153.035517 -27.417948))', 4283)
set @test2Extent = geography::STPolyFromText('POLYGON((149.182384 -21.138607,149.181783 -21.14275,149.1867 -21.1433783,149.187362 -21.139408,149.182384 -21.138607))', 4283)
set @test3Extent = geography::STPolyFromText('POLYGON((153.130231 -26.753869,153.129566 -26.756992,153.13199 -26.756609,153.131733 -26.753831,153.130231 -26.753869))', 4283)
set @test4Extent = geography::STPolyFromText('POLYGON((152.753145 -27.611423,152.753885 -27.613172,152.757018 -27.61244,152.755065 -27.610358,152.753145 -27.611423))', 4283)
declare @iterations int
declare @iterationCount int
declare @start datetime
declare @diff decimal
declare @sumTest1 decimal(10,6)
declare @sumTest2 decimal(10,6)
declare @sumTest3 decimal(10,6)
declare @sumTest4 decimal(10,6)
declare @count int
set @iterations = 100
set @iterationCount = 0
set @sumTest1 = 0
set @sumTest2 = 0
set @sumTest3 = 0
set @sumTest4 = 0
while (@iterationCount < @iterations)
begin
set @start = GETDATE()
-- Execute selection 1
select @count = COUNT(id)
from state_1 with (index(IX_SP_state_1))
where geom.STIntersects(@test1Extent) = 1;
set @diff = DATEDIFF(ms, @start, GETDATE())
-- Add the elapsed time to the total for selection 1
set @sumTest1 = @sumTest1 + convert(decimal(10,6),(@diff/1000))
set @start = GETDATE()
-- Execute selection 2
select @count = COUNT(id)
from state_1 with (index(IX_SP_state_1))
where geom.STIntersects(@test2Extent) = 1;
set @diff = DATEDIFF(ms, @start, GETDATE())
-- Add the elapsed time to the total for selection 2
set @sumTest2 = @sumTest2 + convert(decimal(10,6),(@diff/1000))
set @start = GETDATE()
-- Execute selection 3
select @count = COUNT(id)
from state_1 with (index(IX_SP_state_1))
where geom.STIntersects(@test3Extent) = 1;
set @diff = DATEDIFF(ms, @start, GETDATE())
-- Add the elapsed time to the total for selection 3
set @sumTest3 = @sumTest3 + convert(decimal(10,6),(@diff/1000))
set @start = GETDATE()
-- Execute selection 4
select @count = COUNT(id)
from state_1 with (index(IX_SP_state_1))
where geom.STIntersects(@test4Extent) = 1;
set @diff = DATEDIFF(ms, @start, GETDATE())
-- Add the elapsed time to the total for selection 4
set @sumTest4 = @sumTest4 + convert(decimal(10,6),(@diff/1000))
set @iterationCount = @iterationCount + 1
end
-- Calculate the average times
print 'Test 1 Time: ' + convert(varchar(10), convert(decimal(10,6), (@sumTest1/@iterations))) + ' seconds'
print 'Test 2 Time: ' + convert(varchar(10), convert(decimal(10,6), (@sumTest2/@iterations))) + ' seconds'
print 'Test 3 Time: ' + convert(varchar(10), convert(decimal(10,6), (@sumTest3/@iterations))) + ' seconds'
print 'Test 4 Time: ' + convert(varchar(10), convert(decimal(10,6), (@sumTest4/@iterations))) + ' seconds'
</pre><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: large;"><br />
<b>Index Tuning</b><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: normal;"><br />
This is the script I executed to examine the spatial index efficiency for the Geography based data set.<br />
<br />
The difference between this script and the Geometry based index analysis script was that the extents were defined as Geometry variables, and the instance of the Geometry was created using <i>geometry::STPolyFromText</i>, supplying 0 as the SRID, and the <i>sp_help_spatial_geometry_index</i> stored procedure was used for the Geometry data set.<br />
<br />
</span><br />
<pre class="brush: sql; highlight: [1,2,3,4,7,8,9,10,12,13,14,15]">declare @test1Extent geography
declare @test2Extent geography
declare @test3Extent geography
declare @test4Extent geography
-- Define the selection extents
set @test1Extent = geography::STPolyFromText('POLYGON((153.035517 -27.417948,153.032598 -27.42869,153.049808 -27.431585,153.05161 -27.420271,153.035517 -27.417948))', 4283)
set @test2Extent = geography::STPolyFromText('POLYGON((149.182384 -21.138607,149.181783 -21.14275,149.1867 -21.1433783,149.187362 -21.139408,149.182384 -21.138607))', 4283)
set @test3Extent = geography::STPolyFromText('POLYGON((153.130231 -26.753869,153.129566 -26.756992,153.13199 -26.756609,153.131733 -26.753831,153.130231 -26.753869))', 4283)
set @test4Extent = geography::STPolyFromText('POLYGON((152.753145 -27.611423,152.753885 -27.613172,152.757018 -27.61244,152.755065 -27.610358,152.753145 -27.611423))', 4283)
exec sp_help_spatial_geography_index 'state_1', 'IX_SP_state_1', 1, @test1Extent;
exec sp_help_spatial_geography_index 'state_1', 'IX_SP_state_1', 1, @test2Extent;
exec sp_help_spatial_geography_index 'state_1', 'IX_SP_state_1', 1, @test3Extent;
exec sp_help_spatial_geography_index 'state_1', 'IX_SP_state_1', 1, @test4Extent;
</pre>stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com0tag:blogger.com,1999:blog-392007053337460683.post-42397420217658399852011-04-26T04:30:00.001-07:002011-04-26T05:43:00.138-07:00The Black Art Of Spatial Index Tuning In SQL Server<div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">For my inaugural blog post I have decided to discuss some investigations I have been making recently into tuning spatial indexes in SQL Server.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I have found that while there is a lot of information on <span style="font-style: italic;">teh interwebs</span> about the theory of SQL Server spatial indexes, and the utilities that can be used to tune them, there is very little guidance on what spatial index settings are best suited for different data scenarios, and there is only limited information about the characteristics to consider when tuning these indexes.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As an attempt to provide some guidance I have investigated configuring and tuning spatial indexes on three different spatial data sets which I have discussed as case studies, but before jumping straight into these I think it would be wise to discuss some background aspects to spatial indexing in SQL Server.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><br />
<a name='more'></a><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>SQL Server Spatial Index Theory</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I'm not going to get too much into the theory of spatial indexes in SQL server, or the details of creating spatial indexes because there is a lot of documentation on this subject, but I will be referring to aspects of spatial index theory throughout this post. The best starting point for delving into this area is the SQL Server <a href="http://technet.microsoft.com/en-us/library/bb964712.aspx">Spatial Indexing Overview documentation</a>. Another great source of information is <a href="http://blogs.msdn.com/b/isaac/archive/2008/03/01/basic-multi-level-grids.aspx">Isaac Kunen's Blog</a>.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">A SQL Server spatial index is basically a grid of cells that cover the extent of the spatial data in a table which allows spatial queries to easily filter out prospective records and lookup smaller subsets of records to see if they match the spatial criteria. e.g. If a spatial query is trying to find all the records that intersect a particular rectangle, then a spatial index can improve the performance by only executing the intersection criteria on records that are contained in a grid cell(s) that the criteria rectangle resides in.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Rather than just being a single layer of cells over the extent of the spatial data, the index is actually a layered grid which is four levels deep (see the image from the SQL Server documentation below). Each cell in level 1 of the grid is divided into cells that make up level 2, and so on for level 3 and 4.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-iEcxRefq85o/TbavK1rYrQI/AAAAAAAAABQ/izDJjcrsJbk/s1600/SQLServerSpatialIndexGrids.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="164" src="http://1.bp.blogspot.com/-iEcxRefq85o/TbavK1rYrQI/AAAAAAAAABQ/izDJjcrsJbk/s320/SQLServerSpatialIndexGrids.jpg" width="320" /></span></a></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin: 0in;"></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Each level of the grid can be divided into a different number of cells based on a setting of Low (4x4 or 16 cells), Medium (8x8 or 64 cells), or High (16x16 or 256 cells). This is what controls the granularity of the spatial index.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Geography vs. Geometry</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">An interesting thing to note about the differences between Geography and Geometry data types is that spatial indexes for Geography types do not have a bounding box specified. This means that Geography indexes span the entire globe from -180 to 180 longitude and -90 to 90 latitude. This means that the granularity specified for a Geography index divides the entire globe into the desired number of cells.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Geometry indexes have a bounding box specified, so the administrator can define the grid granularity in conjunction with a known data extent, meaning that the administrator has a lot more control over the equivalent "ground" size of each grid cell.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Because of this limitation I see the Geometry index as having a major advantage over the Geography index. I'm not entirely sure of the reason for this constraint on Geography indexes, perhaps there is an assumption that data sets using geographic coordinates (longitude, latitude) are usually large scale data sets spanning large extents around the earth, but in practice I don't think this is always the case, which will become evident in my last case study.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Index Distribution</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In SQL Server a primary key index has high selectivity because each indexed value is unique, which means that performance is relatively similar across the distribution of keys in the index regardless of the selection criteria.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">It is a different story with non-unique indexes, because there may not be the same level of selectivity across the entire distribution of values in the index, e.g. A column containing surnames of users would undoubtedly have low selectivity when searching for names such as "smith" or "jones", but potentially have higher selectivity when searching for names such as "kerouac" or "zimmerman". These indexes show us that the performance of the index has a high correlation with the criteria we are using to search with. </span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This same consideration needs to be applied when we create spatial indexes. When calculating the desired settings for a spatial index, not only is it important to make sure that the index is granular enough to efficiently index our records, it is also important that we have a rough idea on the range of spatial selections that users will be performing on our data set.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Based on the four levels of index grid, and three settings of granularity of each grid, it can be seen that the resulting number of cells in the grid will be the same if the index is defined as High, Medium, Medium, Medium (HMMM) as it would if the index was defined as MMMH, but the difference in performance will be dependent on the spatial criteria being applied, i.e. A large sized rectangular criteria filter may perform better using the MMMH index over the HMMM index because it may be able to select fewer grid cells in level 1 to analyse before drilling into the level 2 grid cells.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Spatial Index Tuning</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I found a great source of information on spatial index tuning on <a href="http://www.sqlskills.com/BLOGS/BOBB/post/Using-the-spatial-index-diagnostic-stored-procedures.aspx">Bob Beauchemin's Blog</a> which highlights the following stored procedures for investigating spatial index characteristics:</span></div><ul style="direction: ltr; margin-bottom: 0in; margin-left: .75in; margin-top: 0in; unicode-bidi: embed;" type="disc"><li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;"> sp_help_spatial_geography_index</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;"> sp_help_spatial_geometry_index</span></li>
</ul><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This stored procedures accept as input parameters</span></div><ul style="direction: ltr; margin-bottom: 0in; margin-left: .75in; margin-top: 0in; unicode-bidi: embed;" type="disc"><li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Table Name</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Index Name</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Flag indicating whether to return verbose output</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">A geography/geometry instance representing the selection extent (refer to the index distribution discussion above)</span></li>
</ul><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In Bob's Blog he discusses the interesting characteristics being the primary and internal filter efficiency, as well as the statistics on the overall number of rows in the index, broken down as an average number of index rows per row of data.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The primary filter is the spatial index, and the internal filter seems to be an optimised version of the spatial index. The spatial filter's purpose is to reduce the number of times the query's spatial criteria is compared against records in the table, by executing it against cells in the index and looking up the data that resides in these cells.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In this <a href="http://www.sqlskills.com/BLOGS/BOBB/post/Spatial-Index-Diagnostic-Procs-Filter-Output.aspx">post</a> on Bob Beauchemin's Blog he discusses the output values from the tuning stored procedures, showing that the primary index alone will result in selection of a records that may meet the spatial criteria, including a number of false positives, while the optimised internal filter will perform a conservative selection of the bare minimum of records that meet the spatial criteria, and it is the difference between these selections that the query engine will use to compare the spatial criteria against the data records to determine whether they meet the query criteria.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">So the most efficient spatial indexes will have a large percentage of the records being determined from the primary and internal filters (i.e. high efficiency percentages), with only a small number of surplus records needing to be checked individually.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Keep in mind that if we make the index quite granular, i.e. have a large number of small grid cells, we may be able to improve the efficiency percentage of the primary and internal filters, but the index itself will need to analyse a larger number of grid cells to see if those match the query criteria, and once the matching cells are determined, the index will possibly need to look-up the data records from more index leaf nodes.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">From these observations we can see that the perfect spatial index will be defined such that the number of grid cells in the index is minimised, while still providing a granularity of the index grid to allow selections to be made across the entire data set, in such a way that the index filter efficiency will be maximised across the broad range of spatial queries being performed.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Wow, that's a mouthful, breaking that down, basically we want the index to be as efficient as possible while trying to minimise the number of rows in the index.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Index Granularity</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As discussed in the Index Distribution section above, the main consideration for the spatial index settings comes down to having a good idea of the types of spatial queries that will be getting performed. e.g. If the majority of spatial selections are for view extents, and they are mostly higher scale selections, then it might be appropriate to have an index grid density that is relatively sparse compared to the spatial density of the data in the table. If on the other hand you are doing a lot of selections of small numbers of records, perhaps within a buffer of another feature, then you may want to have an index grid density that is fairly similar to the density of your data.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This leads to us needing to know a little more about the statistics of our data. Typical indexes in SQL Server use statistics in their indexes to determine how selective particular data values are across the range of data in the table, so in a sense we are doing something similar in getting to know the distribution of our data to make an informed decision on the settings for our spatial index.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">In my case studies below I performed spatial analysis of my data sets to determine the maximum density of data, and made decisions on my spatial indexes using that information. I did this using a T-SQL script that splits the data extent into a number of blocks, and counts the number of records in each block looking for the maximum number. It then calculates a relative density compared to the extent size using the number of records in the block with the maximum number.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Spatial Index Hints</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Just briefly I wanted to mention that when performing my analysis of spatial indexes in SQL Server I found that I had to always define an index hint to force the spatial queries to use the index I had defined. I have read that this may be due to the version of SQL Server I am using, but I thought it would be useful to make a note of so that anyone having problems with the spatial index not being used can be aware that this might be a cause.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The image below shows my query plan without an index hint, showing that a table scan (in this case the clustered primary key) will be performed to check each record to see if it matches the criteria expression.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-cBfzKwxTySY/Tbavt6fONDI/AAAAAAAAABU/6aJP34M-hr0/s1600/WithNoIndexHint.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="50" src="http://3.bp.blogspot.com/-cBfzKwxTySY/Tbavt6fONDI/AAAAAAAAABU/6aJP34M-hr0/s400/WithNoIndexHint.jpg" width="400" /></span></a></div><div class="separator" style="clear: both; text-align: center;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div class="separator" style="clear: both; text-align: left;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The image below shows the same query with an index hint defined, showing part of the query plan that indicates a spatial index seek will be employed.</span></div><div class="separator" style="clear: both; text-align: left;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-DU8Tzbeamsc/TbavwJaRt7I/AAAAAAAAABY/iPgZ40I3KXw/s1600/WithIndexHint.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="110" src="http://4.bp.blogspot.com/-DU8Tzbeamsc/TbavwJaRt7I/AAAAAAAAABY/iPgZ40I3KXw/s400/WithIndexHint.jpg" width="400" /></span></a></div><div class="separator" style="clear: both; text-align: left;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div class="separator" style="clear: both; text-align: left;"></div><div style="font-size: 12pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: 15px;"><b>Case Studies</b></span><br />
<div style="font-weight: bold;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: 15px; font-weight: normal;"><br />
</span></div><div style="font-weight: bold;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: 15px; font-weight: normal;">I have a background in cadastral surveying, so I have used three data sets that each represent a layer of cadastral land boundaries, as land parcel polygons:</span></div></div><ul style="direction: ltr; margin-bottom: 0in; margin-left: .75in; margin-top: 0in; unicode-bidi: embed;" type="disc"><li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Data Set 1 - Local Municipality - approximately 15km x 15km, with 14000 records</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Data Set 2 - Local Municipality - approximately 40km x 40km, with 66000 records</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Data Set 3 - State - approximately 1700km x 2200km, with 2.5 million records</span></li>
</ul><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">For each of the data sets I created four separate selections in different locations within the data set, and then wrote a script that would execute the four selections a hundred times and then average the time taken to select the records. The number of features selected is not exactly the same for each data set, but is relatively similar enough to make comparisons.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">For each of the data sets, the scenarios that I had in mind were spatial selections of urban or semi-urban land parcels, so three of the four spatial selections in the tests select a relatively small number of records, i.e. akin to selecting the land parcels in a street, or subdivision etc. The fourth selection is like a selection for a view extent, of a number of thousand parcels, i.e. around 1:10000 scale.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Data Set 1</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This data set represents a small regional local government authority. The extent of the data is about 15km x 15km (10 miles). Regardless of the small size of the area, the majority of land parcels are typical residential sized allotments.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The data set was based on a projected map grid datum, so I was able to create each land parcel as a geometry data type. This meant that I was able to define my spatial indexes with a bounding box, and be in more control of the size of the resulting index grid cells.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The data set has an average parcel size of 10000sqm, with 80% of parcels being under 2000sqm (residential land parcel size), with the total area of those 80% of parcels being 0.64% of the area of the bounding box of the data set, and being 6.78% of the total area of all parcels.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This gives you an indication of the relative density of the data, and that the majority of the small spatial selections that will be performed on the data will be on parcels that are very small compared to the extent of the data set.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I calculated the highest area of spatial density to be 1:238400 (comparison to the size of the overall extent of the data set), which is a lot higher than the number of records in the data set (approx 14000).</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I made a decision that I wanted an index grid that would be similar to the highest record density, so I figured that a grid of MLLL, or 64x16x16x16 (equating to 262144 grid cells) would be the ideal, or a permutation of those settings, i.e. LMLL, LLML, LLLM. I also executed my test script against other index configurations as well to test my hypothesis, e.g. LLLL, HLLL, etc.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><span lang="en-AU">My test selections were performed with an </span><span lang="en-US">STIntersects criteria expression with different sized rectangles in different locations in the area. The results being:</span></span></div><div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Test 1 Count: 56 features</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Test 2 Count: 124 features </span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Test 3 Count: 110 features</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Test 4 Count: 7016 features</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The results for each index are shown below, with the average time taken to perform each query, and the filter efficiency of each selection, along with the number of rows in the index.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.8652in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Index Settings</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Index Performance</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.8652in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">No Index</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7472in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.1081s</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7472in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.1091s</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7472in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.1080s</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7472in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.1266s</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.8652in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">MLLL</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 87980</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 264</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 6</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;"> 0.0049s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 94.64%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 67.46%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0072s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 82.25%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 92.53%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0049s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 88.18%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 88.70%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0433s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 88.18%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 88.70%</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.8652in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">LMLL</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 87955</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 263</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 6</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0039s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 94.64%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 67.46%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0082s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 82.25%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 92.53%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0040s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 88.18%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 88.70%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0417s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 88.54%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 98.63%</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.8652in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">LLML</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 87020</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 261</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 6</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0051s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 94.64%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 67.46%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0074s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 78.22%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 89.20%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0050s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 87.27%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 82.08%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0416s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 87.10%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 98.24%</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.8652in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">LLLL</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 53354</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 161</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 3</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0047s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 73.21%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 58.33%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0070s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 79.83%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 93.23%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0042s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 79.09%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 69.18%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0506s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 73.07%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 97.55%</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.8652in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">HLLL</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 157928</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 473</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 11</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0048s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 100%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 71.79%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0103s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 85.48%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 96.12%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0053s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 90%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 94.01%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7069in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0496s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7076in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 95.89%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 98.40%</span></div></td> </tr>
</tbody></table></div></td> </tr>
</tbody></table></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The results show that creating an index with a granularity equivalent to the most dense area of my data resulted in the best performance. The LMLL index provided the best filter efficiency across all four selections, but the timed performance was slightly worse for one of the smaller selections, but this time difference is negligible in the context of the entire query time.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">You can see that the HLLL index actually provided a much better primary and internal filter efficiency than the MLLL permutations, but because this index results in twice the number of rows of the MLLL indices, the performance time is longer, as I discussed in the Spatial Index Tuning section.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Conversely, you can see from the LLLL index that the number of index rows is significantly less, but the primary and internal filter efficiency is worse, because the granularity of the grid cells does not support the queries that we are performing, so the query engine is having to resort to comparing more of the data against the spatial criteria, and hence the efficiency is poor.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">One thing to note about this data set is that it is so small (14000 records) that the performance with no index is still quite good, with the table scan only taking around 0.1s to check each record in the table to see if it matches the spatial criteria (this is why the time is very similar for each test). It is worth noting though that by adding the spatial index the large selection query performance is 3 times faster, and the smaller selections are 20 times faster.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Data Set 2</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This data set represents a medium sized suburban local government authority. The extent of the data is about 40km x 40km (25 miles). The data has a number of medium to high density suburban areas, and there are large portions within the extent of data set that are not utilised as land parcels, leading to the data set having areas of higher density parcels relative to the size of the extent of the data set.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Like the previous data set, this data was based on a projected map grid datum, so I was able to create each land parcel as a geometry data type. This meant that I was able to define my index with a bounding box, and be in more control of the size of the resulting index grid cells.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This data set has an average parcel size of 8000sqm, with 90% of parcels being under 2000sqm, with the total area of those 90% of parcels being 2.78% of the area of the extents of the data set, and being 7.73% of total area of all parcels.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I calculated the highest area of spatial density to be 1:1189200, which is a lot higher than the number of records in the data set (approx 66000).</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Again, basing my decision for the index grid size to be similar to the highest record density, I decided on a grid size of MMLL (64x64x16x16) or HLLL (256x16x16x16) to equate to 1048576 grid cells.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The spatial selection test results are shown below:</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Test 1 Count: 82 features</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Test 2 Count: 167 features </span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Test 3 Count: 116 features</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Test 4 Count: 6226 features</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Index Settings</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Index Performance</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">No Index</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7791in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.4770s </span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7791in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.4755s</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7791in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.4784s </span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7791in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.4958s</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">MMLL</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 274349</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 820</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 4</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0046s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 50%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 84.53%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0072s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 91.01%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 97.66%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0048s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 77.58%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 76.31%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0417s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 84.90%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 96.94%</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">MLML</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 273834</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 818</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 4</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0043s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 50%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 84.53%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0079s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 91.01%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 97.66%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0047s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 77.58%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 76.31%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0421s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 84.86%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 96.90%</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">HLLL</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 274443</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 820</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 4</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0051s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 50%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 84.53%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0076s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 91.01%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 97.66%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0053s</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 77.58%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 76.31%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0403s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 84.90%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 96.94%</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">MMML</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 489475</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 1459</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 7</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0051s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 92.41%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 97.14%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0100s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 86.82%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 95.42%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0054s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 82.92%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 88.17%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0386s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 91.37%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 87.87%</span></div></td> </tr>
</tbody></table></div></td> </tr>
</tbody></table></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As we can see again, by making the index granularity similar to the density of the most dense data in the data set, we have defined the best performing spatial index - MMLL. As I mentioned above, the grid cell granularity of MMLL is the same as HLLL. What is interesting to note is that the primary and internal filter efficiency is exactly the same between MMLL and HLLL, and the number of index rows is very similar as well, but the time taken to perform the larger selections is slightly faster for the HLLL index. It may be that more of the solution may be getting determined by grid levels 1 or 2 and not having to resort to lower grid levels. When using the verbose output setting on the stored procedure, there are results such as Total_Number_Of_ObjectCells_In_Level1_In_Index that will give a breakdown of each grid level, and can be analysed to see what the object/index cell distribution is across each of the levels.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">One of the interesting results in indexes MMLL and HLLL is the efficiency values in test 1. The internal and primary filter efficiency is quite poor. I checked this selection and found that the selection rectangle actually passed through most of the features that were being selected. This means that a high percentage of features needed to be analysed to see if they fit the criteria. One thing we may be able to read into this is that the granularity of the index grid may not be small enough, given than our optimal choice in the first data set resulted in an average index rows per data row of 6, and this choice was 4, perhaps we don't have the granularity we need. Since the selection was so small, and the performance time quite fast anyway, I ignored this result as an outlier, considering it an anomaly of my selection choice.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Another interesting result was the use of the MMML index, which results in twice the number of index rows, but results in an average number of index rows per data row that is closer to our choice from data set 1 - 7. Looking at the filter efficiency we can see that there has been a big improvement for test 1, although the performance time has not improved. Again we can see this as the trade-off between efficiency based on grid granularity, and number of index rows that need to be analysed. If over time we found the MMLL index being inefficient for other selections, we may need to think about using the MMML index, depending on how it was performing.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">It is interesting to note with this data set having 66000 records, that the time to perform a table scan has increased significantly on the time encountered in data set 1, with indexed results being between 10 and 100 times faster.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Data Set 3</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This data set represents the cadastral boundaries for the Australian state of Queensland. For overseas readers, you may recall Queensland being in the news earlier in the year due to severe flooding in the south east, and Cyclone Yasi in the north. Queensland is the 3rd most populated state of Australia, and 2nd in area to Western Australia. The state is 1700km (1000 miles) wide, by 2200km (1400 miles) in length. Most of the population is centred around the south east corner of the state (containing the vast majority of land parcels in the state), with a number of populated regional centres located along the coast.</span></div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-Qrc7UIkpB7A/TbaxvfqykRI/AAAAAAAAABc/sub0E-cFEd0/s1600/AustraliaShowingQueensland.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><img border="0" height="342" src="http://4.bp.blogspot.com/-Qrc7UIkpB7A/TbaxvfqykRI/AAAAAAAAABc/sub0E-cFEd0/s400/AustraliaShowingQueensland.jpg" width="400" /></span></a></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><span lang="en-AU">This data set has an average parcel size of </span><span lang="en-US">654150</span><span lang="en-AU">sqm, with 80% of parcels being under 2000sqm, with the total area of those 90% of parcels being </span><span lang="en-US">0.033</span><span lang="en-AU">% of the area of the extents of the data set, and being </span><span lang="en-US">0.071</span><span lang="en-AU">% of total area of all parcels.</span></span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">To create a spatial index that will cater for the diversity of spatial density for this data set is quite challenging, but as I mentioned in the discussion on Geography vs Geometry, I struck a limitation with this data set that constrained what I could achieve generally. This data set is in geographic coordinates (longitude, latitude), so I am not able to define a bounding box for the spatial index. Instead I was only able to try to make the grid as granular as I could to improve the types of selections I desired.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As with the other two data sets, I calculated the highest area of spatial density to be 1:508659200, which is a lot higher than the number of records in the data set (approx 2.5 million), which if using geometry data I could have created a HHHM (256x256x256x64) 1073741824 grid cell index, but using this granularity for the geography index provided poor efficiency (see below). I was only able to define one level higher in the number of grid cells - HHHH (256x256x256x256) with 4294967296 grid cells, and while this did improve upon the previous settings, was still quite poor performing in terms of efficiency.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div lang="en-US" style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The spatial selection test results are shown below:</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><span lang="en-AU">Test 1 Count: </span><span lang="en-US">79 </span><span lang="en-AU">features</span></span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><span lang="en-AU">Test 2 Count: </span><span lang="en-US">264</span><span lang="en-AU"> features </span></span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><span lang="en-AU">Test 3 Count: </span><span lang="en-US">117</span><span lang="en-AU"> features</span></span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><span lang="en-AU">Test 4 Count: </span><span lang="en-US">6297</span><span lang="en-AU"> features</span></span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Index Settings</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Index Performance</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">No Index</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .859in;"><div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">34.8497s </span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .859in;"><div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">34.9102s </span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .859in;"><div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">34.8761s </span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .859in;"><div lang="en-US" style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">34.9194s</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">HHHL</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 3378393</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 10054</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 0</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0397s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 3.97%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0565s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 8.96%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0343s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 6.92%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.3844s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 31.26%</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">HHHM</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 3882914</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 11554</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 0</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0228s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 6.86%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0197s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 29.01%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0207s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 11.83%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.2183s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 2.79%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 49.22%</span></div></td> </tr>
</tbody></table></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 3.7902in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">HHHH</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Rows 4582733</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Total_Primary_Index_Pages 13637</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Average_Number_Of_Index_Rows_Per_Base_Row 0</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 4.0819in;"><div style="direction: ltr;"><table border="1" cellpadding="0" cellspacing="0" style="border-collapse: collapse; border-color: #A3A3A3; border-style: solid; border-width: 1pt; direction: ltr;" valign="top"><tbody>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 1</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0162s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 11.70%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 2</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0167s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 38.59%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 3</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.0144s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 0%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 19.05%</span></div></td> </tr>
<tr> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .6673in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Test 4</span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: .7416in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">0.1785s </span></div></td> <td style="border-color: #A3A3A3; border-style: solid; border-width: 1pt; padding: 4pt 4pt 4pt 4pt; vertical-align: top; width: 2.7055in;"><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Internal_Filter_Efficiency 9.30%</span></div><div style="margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif; font-size: x-small;">Primary_Filter_Efficiency 61.30%</span></div></td> </tr>
</tbody></table></div></td> </tr>
</tbody></table></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">As discussed above, the use of the Geography data type has constrained our choices with this data set, so HHHH is the natural choice for our index, based on it having the best filter efficiency, and unfortunately not really having any more granular settings at out disposal, i.e. HHHH is the most detailed index grid we can define.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">When performing the selections on this data set with no index, the time taken to perform a table scan was around 35s, so the improvement in using the index was from 200 to 2000 times faster.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Unfortunately the lack of control over the granularity of the grid means that a lot of processing would be occurring in the database to perform each query, so I would expect performance to degrade quickly for this data set in a multi user environment.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">While this data set was interesting to use from an analysis perspective, in an enterprise situation I would expect that the data would be tiled into logical regions, with queries incorporating one or more tiles depending on the size of the query. In this way a tile would act as a kind of preliminary index.</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Conclusion</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Like any system implementation I think that applying a spatial index is something that needs to be reviewed and tweaked over the lifetime of the data set. Depending on the type of data being kept, the data may incrementally change over time, possibly requiring tweaking of the index, but regardless of this, you will also learn more about the queries that users are performing against the data as time goes by, and this knowledge will help you design you spatial index.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Remember that your spatial index will have a profound effect on the performance of your database, especially in a multi user scenario, so the index should be designed well to meet the needs of the queries being performed.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Another suggestion that I haven't spoken of so far is that like typical SQL Server indexes, it is possible to have multiple spatial indexes on the same table, so if necessary you could have an index that is utilised by larger queries, such as view extents at a high scale, and another for granular spatial queries, and then use index hints to swap between the two when performing different operations. This would split the processing and potentially improve the performance in a multi user environment.</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The following are a summary of my main tips from what I have learned:</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><ul style="direction: ltr; margin-bottom: 0in; margin-left: .75in; margin-top: 0in; unicode-bidi: embed;" type="disc"><li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Always define a spatial index, no matter what the settings - don't get hung up on needing the perfect index from the start, as I have shown having any index improves spatial queries over no index at all. As I said you can improve your index over time as you get to understand your users, and your data.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Make sure that your index is being used by checking the query plan.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Determine the density of your data, or at least be able to estimate the density of your desired index grid cells.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Get some information about the types of spatial selections users will be performing - perhaps log all user selections so that you can test your spatial indexes with these queries.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: 'Trebuchet MS', sans-serif; font-size: 11pt;">Use the tuning stored procedures - be objective about filter efficiency vs. number index rows, because an imbalance in either will affect your query performance, and therefore load on the server in a multi user environment.</span></li>
</ul><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span></div><div style="font-size: 11pt; margin-bottom: 0in; margin-left: 0in; margin-right: 0in; margin-top: 0in;"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">My final comment would be that were you have the choice to use Geometry or Geography data, choose Geometry, if only for the control over the spatial index. I see the index limitation with the Geography data type to be a major stumbling block, especially in situations such as my third case study. I have heard that future versions of SQL Server will have the ability to have indexes with deeper grid levels, so this may alleviate the situation I have, so I look forward to seeing this new functionality.</span></div>stoolrossahttp://www.blogger.com/profile/06720166580858034195noreply@blogger.com4