Spatial Tajo : Supporting Spatial Queries on Apache Tajo
Cho, Hyungu
Apache Tajo is a robust big data relational and distributed data warehouse system for Apache Hadoop. Tajo is top level project in Apache Software Foundation, it will be next generated data warehouse system. Tajo supports SQL standard queries, it can't perform queries about spatial objects unfortunately. So I will announce production experience Spatial Tajo extended plug-in for Tajo. The plan of Spatial Tajo is spatial plug-in supporting basic spatial queries, spatial join and spatial indexes, but current progress of this can still perform simple spatial relational functions and a few spatial joins. Spatial indexes will realize.
 Hello everyone, my name is Hyung Woo Jo and I'm doing master course in computer science and at Gunsan National University, the title of my presentation is Special Tazzo Supporting Special Chorus on Tazzo. The presentation will be made in following other Wadi Special Tazzo body for development why I chose part Tazzo plan for the implementation of a plugin, part implemented and part not yet implemented and conclusion. What is special Tazzo? Briefly it is a plugin to provide special queries for Tazzo. In detail it is a plugin to provide and perform calling data set by special queries using SQL in distributed data warehouse system. Tazzo which provides special functions supports special data types for special queries, supports indexing special data and allows the use of rest of the data. Then what would be the development of this plugin? SQL is getting more interest in big data for the last decades. Tazzo supports to analyze special big data too, have increased naturally. I also, those that the volume of the data containing the special information I have tapped it to analyze could come close to big data. The data held by my lab are collection of tweet on Twitter in real time which are not the ones consisting only of special information but I have tapped it to analyze using the data. When I analyzed the data, analysis using Hadoop was a trend and I too conduct analysis using Hadoop. However, I had to use map needles to conduct analysis using Hadoop and often thought that using SQL whenever I conduct an experiment would be convenient. Since it is somewhat difficult for analysis to use map needles which essentially is batch processing the more data, the higher latency becomes. I did research to use existing software solutions. Of course, there are good ones among the existing software and solutions. Transitional special database and DBMS include Oracle special and graph which can be installed to Oracle database as plug-in and my SQL DBMS. Both are satisfactory solutions but are commercial product somewhat difficult to build up a cluster. My university did not provide them separately and solutions that continuously cost or not appropriate option. Using post-UGIS that does not cost seems to be a good method. However, since it is not itself software to analyze big data, it was Hadoop. As for no SQL, there are document-oriented database as MongoDB, KotDB and LithinkDB. I could have used this but I already had structured data so I did not have to use this. HBase was made by modeling after Google's Bigtable but it is somewhat difficult for an analyst to use that. High-based convenient since it does not use SQL but HiveQL. Similar to that, but it was not just that Hive which use map needles would be an appropriate choice. No solutions that use Hadoop include GMS, S3 GIS tools for Hadoop and special Hadoop. S3 GIS tools for Hadoop is close to tools or libraries prepared for analysis using Hadoop while special Hadoop too is close to the form of combination of special plug-in with Hadoop. However, in that it does not cost, despite a little effort should be invested, it seems that performing special queries using Hive or using GMS are maybe most appropriate. However, I finally decided to prepare plug-in that allows Tazo, free-end open source software which low latency to perform special queries by myself. Then, why I choose Tazo? There are features. First I will introduce Tazo before I will speak about why I choose Tazo. First Tazo is robust big data relational and distributed data warehouse system for Hadoop. Tazo is designed for low latency and scalable ad hoc queries, or any obligation and extract transform load process on large dataset stored on Hadoop Distribute 5 system and other data sources. By supporting SQL standard and leveraging advanced database techniques, Tazo allows direct control of distributed execution and data flow across a variety of query evolution strategies and optimization opportunities. Then, why did I choose Tazo? There are a few features that become the reason for my choice. Since Tazo is designed to run on Hadoop, using Hadoop can elevate my cursor about data distribution and also supports Amazon S3. Tazo has a functional external table so it can bring existing files and make queries. It does not support updated syntax like HBase or Hive, but it can override. Tazo supports SQL standard and does not use mem news so it is faster than batch processing using mem news, has felt torrents and supports dynamic scheduling for long-running queries. Tazo is convenient in terms of installation, construction and operation. Of course, it is a project still growing so it is certainly difficult to solve when it fails. If you need, you can implement it yourself and attach it in the form of a program. Tazo, as described above, has managed and demelch as compared to solutions or software in the example, but it is free and open source software so I decided to implement it on Tazo, which was judged to be appropriate considering the characteristics introduced. The plan for implementation of the plugin is broadly decided into four steps. First, for the implementation of special functions for special queries, I decided to implement 11 basic functions including distance and equals and the function for the conversion of special data types. Second, the additional types of special data, which means the custom type containing special information like point line string. Third, KNN query, which means the implementation for smooth performance of KNN after the implementation of special functions. Fourth, indexing special data, which means the implementation of indexed data allows retrieval of necessary data in making special queries. Parts that had been implemented so far include special functions, KNN query, and special data indexing. For special functions, I implemented most of the primary functions using JTS. I carried out KNN queries using the special functions implemented, which are working smoothly. For special data indexing, following the operation methods of index in Tazo and special Hadoop, I implemented a two-level R3 using source tile recursive. The two-level R3 has two forms of global index and local indexes as shown in the picture. The process of building the two-level R3 indexes as follows. First, Tazo of course divides the stored data set by each area using the STR. Second, they build local indexes for each area. Third, they extract only the data on a certain level from each of the local indexes and build a global index using them. The process leading the two-level R3 indexes as follows. First, Tazo of course leads the global index and finds such keys. Second, they find local indexes corresponding to the search keys. Third, they find the search keys in the local indexes. Last three, they read the data directly from the storage and construct tuples. Parts that are not yet implemented include special data types, special functions, K and the query, special data index, and modularization. Special data types are currently not implemented, so there is inconvenience that special functions have to use primitive types. I'm going to resolve the inconvenience of special functions once special data types are implemented. First, as for special functions, I'm going to implement the functions as length area centroids that are not implemented and then optimize each of the special functions and K and the queries. Next, as for the indexing of special data, I'm going to implement quadtree or KDTree using Jewish as well as R3. Lastly, I'm going to modularize the plugin currently. It is combined since it is difficult to separate it from Tazo, but after going through the final modularization in the plugin, it will be distributed in the form of plugins. Conclusion. What is special Tazo? It is a plugin to provide special queries for Tazo. What is Mori for development? I began developing it as I wanted to, as analysis to SQL in distributed data warehouse system instead of using mem news or similar batch processing. Why did I choose Apache Tazo? I did so because I can use it without a great concern about the distribution storage, its support, SQL syntax, especially mem news guarantees, virtual reliance, and above all, I can implement and attach it in the form of plugin myself if necessary. A plan for implementation is as follows. As an overall plan for implementation, I'm going to implement special functions for special queries and special data types and allow learning KNN queries and supporting the function of indexing the special data. The current status is as follows. The patch implemented so far includes most special functions, the learning of KNN queries, and support for indexing through implementing two level R3. The parts not yet implemented include special data types, the remaining special functions, the optimization of special functions and special queries, another method of indexing the special data and the modulation of plugins. For today presentation, I mainly referred to document and source code of Apache Tazo, special hardware and post-jsc and etc. And there are books or website for information about contents as well. For Q&A, please email your questions to the email address here and I will answer them in detail. Thank you for listening to my presentation. Any questions? Have you looked at lib spatial index for doing your indexing rather than trying to do the indexing on your own? Lib spatial index, it's a C library for doing R3 KD3 indexing? No. Any comments or questions? I think how long to implement all functions? Not to plan. I cannot plan. I don't know. Sorry. Any other questions or comments? Okay, thank you.
 Hello everyone, my name is Hyung Woo Jo and I'm doing master course in computer science  and at Gunsan National University, the title of my presentation is Special Tazzo Supporting  Special Chorus on Tazzo.  The presentation will be made in following other Wadi Special Tazzo body for development  why I chose part Tazzo plan for the implementation of a plugin, part implemented and part not  yet implemented and conclusion.  What is special Tazzo?  Briefly it is a plugin to provide special queries for Tazzo.  In detail it is a plugin to provide and perform calling data set by special queries using  SQL in distributed data warehouse system.  Tazzo which provides special functions supports special data types for special queries, supports  indexing special data and allows the use of rest of the data.  Then what would be the development of this plugin?  SQL is getting more interest in big data for the last decades.  Tazzo supports to analyze special big data too, have increased naturally.  I also, those that the volume of the data containing the special information I have  tapped it to analyze could come close to big data.  The data held by my lab are collection of tweet on Twitter in real time which are not  the ones consisting only of special information but I have tapped it to analyze using the  data.  When I analyzed the data, analysis using Hadoop was a trend and I too conduct analysis  using Hadoop.  However, I had to use map needles to conduct analysis using Hadoop and often thought that  using SQL whenever I conduct an experiment would be convenient.  Since it is somewhat difficult for analysis to use map needles which essentially is batch  processing the more data, the higher latency becomes.  I did research to use existing software solutions.  Of course, there are good ones among the existing software and solutions.  Transitional special database and DBMS include Oracle special and graph which can be installed  to Oracle database as plug-in and my SQL DBMS.  Both are satisfactory solutions but are commercial product somewhat difficult to build up a cluster.  My university did not provide them separately and solutions that continuously cost or not  appropriate option.  Using post-UGIS that does not cost seems to be a good method.  However, since it is not itself software to analyze big data, it was Hadoop.  As for no SQL, there are document-oriented database as MongoDB, KotDB and LithinkDB.  I could have used this but I already had structured data so I did not have to use this.  HBase was made by modeling after Google's Bigtable but it is somewhat difficult for an  analyst to use that.  High-based convenient since it does not use SQL but HiveQL.  Similar to that, but it was not just that Hive which use map needles would be an appropriate  choice.  No solutions that use Hadoop include GMS, S3 GIS tools for Hadoop and special Hadoop.  S3 GIS tools for Hadoop is close to tools or libraries prepared for analysis using Hadoop  while special Hadoop too is close to the form of combination of special plug-in with Hadoop.  However, in that it does not cost, despite a little effort should be invested, it seems  that performing special queries using Hive or using GMS are maybe most appropriate.  However, I finally decided to prepare plug-in that allows Tazo, free-end open source software  which low latency to perform special queries by myself.  Then, why I choose Tazo?  There are features.  First I will introduce Tazo before I will speak about why I choose Tazo.  First Tazo is robust big data relational and distributed data warehouse system for Hadoop.  Tazo is designed for low latency and scalable ad hoc queries, or any obligation and extract  transform load process on large dataset stored on Hadoop Distribute 5 system and other data  sources.  By supporting SQL standard and leveraging advanced database techniques, Tazo allows direct control  of distributed execution and data flow across a variety of query evolution strategies and  optimization opportunities.  Then, why did I choose Tazo?  There are a few features that become the reason for my choice.  Since Tazo is designed to run on Hadoop, using Hadoop can elevate my cursor about data distribution  and also supports Amazon S3.  Tazo has a functional external table so it can bring existing files and make queries.  It does not support updated syntax like HBase or Hive, but it can override.  Tazo supports SQL standard and does not use mem news so it is faster than batch processing  using mem news, has felt torrents and supports dynamic scheduling for long-running queries.  Tazo is convenient in terms of installation, construction and operation.  Of course, it is a project still growing so it is certainly difficult to solve when it fails.  If you need, you can implement it yourself and attach it in the form of a program.  Tazo, as described above, has managed and demelch as compared to solutions or software in the  example, but it is free and open source software so I decided to implement it on Tazo, which  was judged to be appropriate considering the characteristics introduced.  The plan for implementation of the plugin is broadly decided into four steps.  First, for the implementation of special functions for special queries, I decided to implement  11 basic functions including distance and equals and the function for the conversion of special  data types.  Second, the additional types of special data, which means the custom type containing special  information like point line string.  Third, KNN query, which means the implementation for smooth performance of KNN after the  implementation of special functions.  Fourth, indexing special data, which means the implementation of indexed data allows  retrieval of necessary data in making special queries.  Parts that had been implemented so far include special functions, KNN query, and special data  indexing.  For special functions, I implemented most of the primary functions using JTS.  I carried out KNN queries using the special functions implemented, which are working smoothly.  For special data indexing, following the operation methods of index in Tazo and special Hadoop,  I implemented a two-level R3 using source tile recursive.  The two-level R3 has two forms of global index and local indexes as shown in the picture.  The process of building the two-level R3 indexes as follows.  First, Tazo of course divides the stored data set by each area using the STR.  Second, they build local indexes for each area.  Third, they extract only the data on a certain level from each of the local indexes and build  a global index using them.  The process leading the two-level R3 indexes as follows.  First, Tazo of course leads the global index and finds such keys.  Second, they find local indexes corresponding to the search keys.  Third, they find the search keys in the local indexes.  Last three, they read the data directly from the storage and construct tuples.  Parts that are not yet implemented include special data types, special functions,  K and the query, special data index, and modularization.  Special data types are currently not implemented, so there is inconvenience that special functions  have to use primitive types.  I'm going to resolve the inconvenience of special functions once special data types are implemented.  First, as for special functions, I'm going to implement the functions as length area  centroids that are not implemented and then optimize each of the special functions and K and the queries.  Next, as for the indexing of special data, I'm going to implement quadtree or KDTree  using Jewish as well as R3.  Lastly, I'm going to modularize the plugin currently.  It is combined since it is difficult to separate it from Tazo, but after going through the final  modularization in the plugin, it will be distributed in the form of plugins.  Conclusion.  What is special Tazo?  It is a plugin to provide special queries for Tazo.  What is Mori for development?  I began developing it as I wanted to, as analysis to SQL in distributed data warehouse system  instead of using mem news or similar batch processing.  Why did I choose Apache Tazo?  I did so because I can use it without a great concern about the distribution storage, its support,  SQL syntax, especially mem news guarantees, virtual reliance, and above all, I can implement  and attach it in the form of plugin myself if necessary.  A plan for implementation is as follows. As an overall plan for implementation, I'm going to implement  special functions for special queries and special data types and allow learning KNN queries  and supporting the function of indexing the special data.  The current status is as follows. The patch implemented so far includes most special functions,  the learning of KNN queries, and support for indexing through implementing two level R3.  The parts not yet implemented include special data types, the remaining special functions,  the optimization of special functions and special queries, another method of indexing  the special data and the modulation of plugins.  For today presentation, I mainly referred to document and source code of Apache Tazo,  special hardware and post-jsc and etc. And there are books or website for information about contents as well.  For Q&A, please email your questions to the email address here and I will answer them in detail.  Thank you for listening to my presentation.  Any questions?  Have you looked at lib spatial index for doing your indexing rather than trying to do the indexing on your own?  Lib spatial index, it's a C library for doing R3 KD3 indexing?  No.  Any comments or questions?  I think how long to implement all functions?  Not to plan.  I cannot plan.  I don't know. Sorry.  Any other questions or comments?  Okay, thank you.
PT3-10_0_95.png PT3-10_96_1007.png PT3-10_1008_1055.png PT3-10_1056_1475.png PT3-10_1476_1667.png PT3-10_1668_2579.png PT3-10_2580_2663.png PT3-10_2664_2927.png PT3-10_2928_4991.png PT3-10_4992_9119.png PT3-10_9120_13655.png PT3-10_13656_15527.png PT3-10_15528_18623.png PT3-10_18624_20747.png PT3-10_20748_21923.png PT3-10_21924_22427.png PT3-10_22428_22931.png PT3-10_22932_23171.png PT3-10_23172_23219.png PT3-10_23220_23699.png PT3-10_23700_23999.png PT3-10_24000_24155.png PT3-10_24156_26363.png PT3-10_26364_26457.png
Hello everyone, my name is Hyung Woo Jo and I'm doing master course in computer science and at Gunsan National University, the title of my presentation is Special Tazzo Supporting Special Chorus on Tazzo. The presentation will be made in following other Wadi Special Tazzo body for development why I chose part Tazzo plan for the implementation of a plugin, part implemented and part not yet implemented and conclusion. What is special Tazzo? Briefly it is a plugin to provide special queries for Tazzo. In detail it is a plugin to provide and perform calling data set by special queries using SQL in distributed data warehouse system. Tazzo which provides special functions supports special data types for special queries, supports indexing special data and allows the use of rest of the data. Then what would be the development of this plugin? SQL is getting more interest in big data for the last decades. Tazzo supports to analyze special big data too, have increased naturally. I also, those that the volume of the data containing the special information I have tapped it to analyze could come close to big data. The data held by my lab are collection of tweet on Twitter in real time which are not the ones consisting only of special information but I have tapped it to analyze using the data. When I analyzed the data, analysis using Hadoop was a trend and I too conduct analysis using Hadoop. However, I had to use map needles to conduct analysis using Hadoop and often thought that using SQL whenever I conduct an experiment would be convenient. Since it is somewhat difficult for analysis to use map needles which essentially is batch processing the more data, the higher latency becomes. I did research to use existing software solutions. Of course, there are good ones among the existing software and solutions. Transitional special database and DBMS include Oracle special and graph which can be installed to Oracle database as plug-in and my SQL DBMS. Both are satisfactory solutions but are commercial product somewhat difficult to build up a cluster. My university did not provide them separately and solutions that continuously cost or not appropriate option. Using post-UGIS that does not cost seems to be a good method. However, since it is not itself software to analyze big data, it was Hadoop. As for no SQL, there are document-oriented database as MongoDB, KotDB and LithinkDB. I could have used this but I already had structured data so I did not have to use this. HBase was made by modeling after Google's Bigtable but it is somewhat difficult for an analyst to use that. High-based convenient since it does not use SQL but HiveQL. Similar to that, but it was not just that Hive which use map needles would be an appropriate choice. No solutions that use Hadoop include GMS, S3 GIS tools for Hadoop and special Hadoop. S3 GIS tools for Hadoop is close to tools or libraries prepared for analysis using Hadoop while special Hadoop too is close to the form of combination of special plug-in with Hadoop. However, in that it does not cost, despite a little effort should be invested, it seems that performing special queries using Hive or using GMS are maybe most appropriate. However, I finally decided to prepare plug-in that allows Tazo, free-end open source software which low latency to perform special queries by myself. Then, why I choose Tazo? There are features. First I will introduce Tazo before I will speak about why I choose Tazo. First Tazo is robust big data relational and distributed data warehouse system for Hadoop. Tazo is designed for low latency and scalable ad hoc queries, or any obligation and extract transform load process on large dataset stored on Hadoop Distribute 5 system and other data sources. By supporting SQL standard and leveraging advanced database techniques, Tazo allows direct control of distributed execution and data flow across a variety of query evolution strategies and optimization opportunities. Then, why did I choose Tazo? There are a few features that become the reason for my choice. Since Tazo is designed to run on Hadoop, using Hadoop can elevate my cursor about data distribution and also supports Amazon S3. Tazo has a functional external table so it can bring existing files and make queries. It does not support updated syntax like HBase or Hive, but it can override. Tazo supports SQL standard and does not use mem news so it is faster than batch processing using mem news, has felt torrents and supports dynamic scheduling for long-running queries. Tazo is convenient in terms of installation, construction and operation. Of course, it is a project still growing so it is certainly difficult to solve when it fails. If you need, you can implement it yourself and attach it in the form of a program. Tazo, as described above, has managed and demelch as compared to solutions or software in the example, but it is free and open source software so I decided to implement it on Tazo, which was judged to be appropriate considering the characteristics introduced. The plan for implementation of the plugin is broadly decided into four steps. First, for the implementation of special functions for special queries, I decided to implement 11 basic functions including distance and equals and the function for the conversion of special data types. Second, the additional types of special data, which means the custom type containing special information like point line string. Third, KNN query, which means the implementation for smooth performance of KNN after the implementation of special functions. Fourth, indexing special data, which means the implementation of indexed data allows retrieval of necessary data in making special queries. Parts that had been implemented so far include special functions, KNN query, and special data indexing. For special functions, I implemented most of the primary functions using JTS. I carried out KNN queries using the special functions implemented, which are working smoothly. For special data indexing, following the operation methods of index in Tazo and special Hadoop, I implemented a two-level R3 using source tile recursive. The two-level R3 has two forms of global index and local indexes as shown in the picture. The process of building the two-level R3 indexes as follows. First, Tazo of course divides the stored data set by each area using the STR. Second, they build local indexes for each area. Third, they extract only the data on a certain level from each of the local indexes and build a global index using them. The process leading the two-level R3 indexes as follows. First, Tazo of course leads the global index and finds such keys. Second, they find local indexes corresponding to the search keys. Third, they find the search keys in the local indexes. Last three, they read the data directly from the storage and construct tuples. Parts that are not yet implemented include special data types, special functions, K and the query, special data index, and modularization. Special data types are currently not implemented, so there is inconvenience that special functions have to use primitive types. I'm going to resolve the inconvenience of special functions once special data types are implemented. First, as for special functions, I'm going to implement the functions as length area centroids that are not implemented and then optimize each of the special functions and K and the queries. Next, as for the indexing of special data, I'm going to implement quadtree or KDTree using Jewish as well as R3. Lastly, I'm going to modularize the plugin currently. It is combined since it is difficult to separate it from Tazo, but after going through the final modularization in the plugin, it will be distributed in the form of plugins. Conclusion. What is special Tazo? It is a plugin to provide special queries for Tazo. What is Mori for development? I began developing it as I wanted to, as analysis to SQL in distributed data warehouse system instead of using mem news or similar batch processing. Why did I choose Apache Tazo? I did so because I can use it without a great concern about the distribution storage, its support, SQL syntax, especially mem news guarantees, virtual reliance, and above all, I can implement and attach it in the form of plugin myself if necessary. A plan for implementation is as follows. As an overall plan for implementation, I'm going to implement special functions for special queries and special data types and allow learning KNN queries and supporting the function of indexing the special data. The current status is as follows. The patch implemented so far includes most special functions, the learning of KNN queries, and support for indexing through implementing two level R3. The parts not yet implemented include special data types, the remaining special functions, the optimization of special functions and special queries, another method of indexing the special data and the modulation of plugins. For today presentation, I mainly referred to document and source code of Apache Tazo, special hardware and post-jsc and etc. And there are books or website for information about contents as well. For Q&A, please email your questions to the email address here and I will answer them in detail. Thank you for listening to my presentation. Any questions? Have you looked at lib spatial index for doing your indexing rather than trying to do the indexing on your own? Lib spatial index, it's a C library for doing R3 KD3 indexing? No. Any comments or questions? I think how long to implement all functions? Not to plan. I cannot plan. I don't know. Sorry. Any other questions or comments? Okay, thank you.
would be the development of this plugin? SQL is about why I choose Tazo. First Tazo is robust big data relational and distributed data warehouse system for Hadoop. Tazo is designed for low Hive, but it can override. Tazo supports SQL course, it is a project still growing so it is introduced. The plan for implementation of the modularization in the plugin, it will be functions for special queries and special data