text: We are in the Golden Age of Data. For those of us on the front-lines, it doesn’t feel that way. Every step forward this technology takes, the need for deeper analytics takes two. We're constantly catching up.
  Necessity is the mother of invention, and mum’s been very busy. In the past decade we have seen ground-breaking after ground-breaking technological advancements in data processing technology and in 2015 we have arrived in the age in the Spark.
  These were the dark days of inflexible batch processes that produced inflexible canned reports. It felt like you were in a full body cast. This was probably for the best, because otherwise you might just hit your head against the wall.
  The most common approach to data platforms was to use batch Extract-Transform-Load (ETL) processes to transform incoming data into ready-made chunks that would be bulk-loaded into a data warehouse.
  For low-latency queries, data warehouses were complemented with OLAP Cubes. Inflexibility was immense and most data platforms were on a daily schedule with simple changes in business logic resulting in weeks if not months of cascading technical work.
  This was golden age for commercial vendors yet difficult times for enterprises managing even small amounts of data. Businesses needed to move quicker, and these bulky processes were slowing businesses down. Businesses had ad-hoc questions that were beyond the reaches of canned reports and this was the main catalyst of change.
  From 2006 to 2009, Multiple Parallel-Processors (MPP) databases brought scalability and ridiculous speed to the Data Warehouse and made OLAP cubes obsolete, resulting in a consolidation of the stack. MPP vendors such as Greenplum, Netezza, and Vertica rose to dominance and former industry leaders responded with their own solutions such as Oracle's Exadata; Teradata had been playing in this space already. Even in the nascent days of MPPs change was occurring: business needs were evolving and the ugly beast of semi-structured data was rearing it’s head.
  Semi-structured data started flooding data platforms with the rise of NoSQL databases like MongoDB and the increased need to analyze RESTful and SOAP API log and response data. The liberation of developers from strict schemas conflicted directly with the foundation of relational databases.
  Companies wanted to analyze these new data sources and the pressure to massage semi-structured and unstructured data into a strict schema put a massive strain on ETL processes.
  Beyond this, there was another fundamental problem: companies were accumulating and collecting data that they could not fit into a relational data model because they did not yet know how they were going to use it. The restrictiveness of needing a data model a priori meant that truly exploratory analytics to unlock hidden value in data was still nascent.
  Hadoop came onto the scene and gave businesses a place to dump ANY type of data and allow proto-data-scientists to poke sticks at it, relieving the pressure on MPPs to be everything to everyone.
  In 2009, Cloudera released the first version of their Hadoop distribution, and by 2010 Hadoop was starting to become a household name in mainstream enterprises. The losers in this shift quickly became ETL tools, which were displaced in droves by Hadoop which could do all of that heavy lifting as well.
  The best practice architecture quickly became Hadoop + MPP with Hadoop becoming de-facto ETL platform transforming data to be loaded to MPP databases. Whatever could not be shoved into the MPP database was analyzed in Hadoop — albeit much more slowly through tools like Hive and Pig.
  This was a good point of stability, but business needs were changing again: increased data volumes put massive pressure on the MPPs to load data quickly, and the data with the most value to extract shifted from being the structured data to the semi-structured data that was sitting in Hadoop. This meant that executing analytics directly on Hadoop became critical.
  MPP vendors put out "Hadoop connectors" that could pull data from Hadoop into the MPP for processing -- but this had a very negative impact on performance as compute needed to be close to storage. There was another simultaneous shift -- the need to analyze streams of data in near real-time.
  The solution was starting to become clear: the world needed a system that could take in vast amounts of data and perform batch and streaming operations without flinching. Nathan Marz created the concept of the Lambda architecture based on his work at Twitter.
  In Lambda, there is a "batch layer" in Hadoop which does safe, reliable processing of data, and a "speed layer" which does real-time stream processing and does needs not be super-reliable. The Lambda stack is traditionally created with Kafka + Storm as the speed layer and Hadoop as the batch layer.
  The stack would process the same data in both layers, the speed layer reacting as soon as the data was created and the batch layer following up with more reliable, hardened processing. The Lambda architecture’s main issues come from its complexity. Jay Kreps did a great job of exploring this in his blog post.
  Data engineers needed to implement business logic in two places, with two frameworks, managing two infrastructures, and composing a single view of the data from two sources. The market and community reacted to these shortcomings—there was Summingbird which provided a common API for the speed and batch layer; and then there was Hortonworks included Storm in their Hadoop distribution, unifying infrastructure and management to some degree.
  Lambda is very relevant today and has a lot of great properties, but many data engineers kept their eyes fixed on the horizons, looking for changes in technology that would make building a stack less like rocket science.
  Today, Spark is dominating mindshare in the data engineering community. Even as an emerging technology Spark addresses a lot of the issues discussed in the previous sections:
  Unified API and infrastructure for Spark & Spark Streaming. Lambda-style architectures are much more approachable.
  A data engineer-friendly API. Spark arrives on the scene with the ease of use that Hadoop DSLs eventually evolved with frameworks like Scalding.
  Tiered storage. Spark can cache data in memory, local disk or back to HDFS. This allows developers to optimize their applications even further.
  Apache Tez deserves a mention because it is a framework that overlaps with Spark in terms of being able to construct an Direct Acyclical Graph (DAG) that distributes and executes processing across tiered storage. Tez was developed to plug into existing frameworks that have data engineer-friendly APIs such as Pig, Hive and Cascading.
  It was not meant to be used directly by data engineers because it’s APIs are so low-level. As a result, it has not received the same amount of attention in the community, but Hortonworks is responding with the Spark-on-Tez project which should be exciting to watch.
  Last but not least, even though Spark can live outside of Hadoop, the two are intertwined since most of the data Spark will be processing lives in HDFS. The gravity of HDFS is immense as it builds a “data fabric” unto which analytic applications are built, and cannot be ignored. Spark will need to continue to build upon and improve its Hadoop ecosystem support.
  OLAP cubes have made a comeback in the Hadoop ecosystem with the creation of aggregates pushed into HBase, as well as with projects like Kylin and commercial products like Platfora.
  At Localytics we are continuously innovating how we approach our data platform. We are ingesting 2B data points a day into our stack built with SQS, Scala+Akka, DynamoDB, EMR and MPP databases. As our business has evolved, we needed to introduce another cornerstone piece, and we chose Spark.
  Last week, we deployed into production our first Spark-based product, Auto-Profiles, that is feeding off of our firehose of data. We’ve been really pleased with Spark thus far, and we are going to build more.
  The public is invited to cheer on students in 6th to 12th grades as they participate in our first ever Tween/Teen Pitch Night. Ten finalists have been selected from a pool of applications to develop their marketing skills as they pitch their designs for innovative and purposeful products to a panel of expert judges.
  Cash prizes for this “shark tank” style program will be awarded for the 3 best pitches in each of 2 age categories - 6th-8th grades and 9th-12th.
  This event is one of many planned for the City of Hudson’s celebration of Global Entrepreneurship Week (GEW), returning for the fifth consecutive year November 11th - 17th. “Pitch Night” and GEW 2018 are generously supported by the Burton D. Morgan Foundation. Sponsored by The Burton D. Morgan Foundation.
  Calling all budding artists in Grades 5 & up: learn about an artist's style and then try it for yourself! Workshop will meet on Thursday, November 15th from 4:30-5:30 pm in the Teen Room. Registration is required - for tweens in Grades 5 and up.
  Toddlers love getting creative and messy! We let small hands take the lead in this artsy storytime where children ages 16 months to 3 years and their caregivers will create art using a variety of materials.
  Come join us for creative, open-ended art using a variety of tactile materials such as paint, glitter, glue and more in this artsy story time for preschooler’s and their caregiver.
  Teens come #ridethestorm here with us for our Fortnite Friday Frenzy! Computers and the PS4 are first come, first served. Feel free to bring your own devices and your friends! #letthedarknessrise
  Magician Gordon Russ returns to the Hudson Library, this time with his “Totally Gross Magic Show!” Gordon’s alter ego, Dr. Ooze, creates magic at its grossest! From slimey green stuff to foot fungus to strange bodily noises, this show is a little stinky, a little booger-y, and a whole lot of silly fun! Warning: this show is rated PG … Pretty Gross!
  Come enjoy classic hits made famous by the likes of Tom Petty, Neil Young, The Byrds, Crosby, Stills & Nash, Tommy James and the Shondells and many more. The band will feature a tribute to the British Invasion with covers of hits by the Beatles, Herman’s Hermits, The Dave Clark Five, The Who and others. The band’s tight vocal harmonies and authentic instruments are a blast from the past and sure to delight audiences of all ages. No registration is required.
  Professional flower arranger Helen Grannetino returns to the Hudson Library & Historical Society to help you design a beautiful floral arrangement in a favorite mug for your Thanksgiving celebration. Please be sure to bring your own mug for this class. No previous flower arranging experience is necessary to attend. Grannetino will also demonstrate some great ideas for entertaining including ideas for punch bowl and buffet decorations.
  Space is limited. Registration is required in person at the second floor reference desk. A nonrefundable $5 materials fee per person must be paid at the time of registration.
  Back by popular demand, members of the Center for Health, Happiness, and Chan, a local non-religious, non-profit organization, will lead morning meditation classes. Learn how to pay more attention to your existence in the present moment and use the breath as the anchor point to calm and balance the mind. Beginners welcome at every class!
  Calling all young artists: come learn about art and create some too! Workshop will take place in the Teen Room on Monday, November 19th from 4:30 -5:15 pm. Registration is required - for children entering Grades K-2.
  Got to be honest that I am super nervous today! I will have a thesis meeting with my supervisor and an interview for a possible future job at her office too! Wish me luck 🙂
  But some good news in the morning, an abstract paper of mine got accepted for the Passive Low Energy Architecture 2016 (PLEA 2016), a conference that will be held in Los Angeles next year. So a good start for this big day! 😀
  This entry was posted in Around campus, Student life, Studies and tagged Conference, EEBD, Interview, Job, Master Thesis. Bookmark the permalink.	
  The Ph.D. program in Transportation Planning and Engineering pairs students with world experts in apprenticeships to solving real problems facing society. You can become researchers within the Connected Cities for Smart Mobility toward Accessible and Resilient Transportation (C2SMART) in the Tandon School of Engineering, where you get to work closely with researchers from other disciplines to conduct basic and applied research funded by the National Science Foundation, U.S. DOT, and other agencies.
  Graduates of this program have successfully pursued careers in academia and technical leadership positions in government and industry around the world. Having a Ph.D. from this program opens opportunities to work with agencies or nonprofit organizations like the World Bank, the U.S. Department of Transportation, or RAND Corp; or with top technology firms like IBM, Google, Amazon, or Ford.
  Several faculty members pursue research funded by various agencies and companies, and in many cases may be able to provide full funding support to interested students.
  Admission to this program requires an MS in Transportation Planning and Engineering or equivalent. Applicants from related backgrounds are encouraged to apply or to contact the faculty members to discuss their interests, whether they are in industrial engineering, operations research, electrical engineering, computer science, applied mathematics, statistics, or economics.
  51 credits of graduate work (not including the Ph.D. dissertation) in relevant major and minor areas of study beyond the bachelor’s degree, with an average grade of B or better (cumulative average of 3.0 or better on a 0-4 scale).
  Completion and successful defense of a 24-credit dissertation related to the major area of study. Dissertations must consist of original research that meaningfully advances the state-of-art in the subject area of the research and should result in the publication of at least 1 paper in a strictly peer-reviewed technical journal related to the subject. A grade of B or better must be achieved for the dissertation.
  Completion of 2 minor areas of study, each consisting of between 9 and 12 credits of graduate work. At least 1 minor area must be outside the transportation area.
  Residency requirements for the Ph.D. in Transportation Planning and Engineering include the 24-credit dissertation, plus a minimum of 9 credits of applicable graduate course work taken at the School of Engineering.
  In meeting the 51-credit course requirement, you must satisfy all requirements for the major and minor areas selected or their equivalent. In satisfying these basic Ph.D. requirements, you must also meet 1 of 2 conditions:
  39 credits of approved graduate coursework, not including individual guided studies (readings, projects, theses, etc.) beyond the bachelor’s degree, with an average grade of B or better (cumulative average of 3.0 or better on a 0-4 scale).
  21 credits of related graduate coursework beyond the master’s degree, with an average grade of B or better (cumulative average of 3.0 or better on a 0-4 scale).
  Satisfying condition 2 requires that the department accept your MS degree in toto without regard to its specific content. This requires a recommendation from the department’s Graduate Committee and the approval of the department head.
  Details about doctoral committees, qualifying examinations, and dissertation policies and procedures can be found in the School of Engineering Bulletin.
  Although a defensive structure has existed on this site since Roman times, Robert Curthose (William the Conqueror’s eldest son) built a new castle here in 1080, calling it New Castle upon Tyne. The castle isn’t new anymore but the name stuck.
  In the 12th century, Henry II built the stone castle keep and Henry III added the Black Gate between 1247 and 1250.
  The keep and the Black Gate are the two remaining parts of the castle and both structures comprise the visitor attraction, Newcastle Castle.
  = a person, especially a politician, who gets money, possessions, etc. from other people without doing much to earn them.