text: Learn the 3 + 1 Vs and how they affect choice in data processing design in this article by Anuj Kumar, a Senior Enterprise Architect with FireEye, a Cybersecurity Service Provider and the author of Easy Test
  Data usually exhibits three characteristics when it comes to designing the data collection system. The industry calls it the 3 V’s of data. The biggest challenge in data processing is to actually understand the three Vs of the data-intensive system and their effect on the overall approach to the design of the data processing pipeline.
  The three Vs stand for the velocity of the data, the volume of the data, and the variety of the data. The outcome from the data processing system is generally the fourth V of the equation, that is, the value of the data. Some experts add more Vs into this equation. For example, data veracity (depicting abnormalities in the data), data validity (the data represents what it is intended to represent), and data volatility (expressing
  in loose terms the importance of data over a period of time). While they all are important, they are more or less covered with the basic three Vs of data.
  When talking about data-intensive systems, it’s safe to assume that the volume of the data will be huge. Thus, the data processing system that you will design should be capable of handling large volumes of data at a regular pace. In today’s world, if you want to make sense out of your data, the data against which you are trying to find value is probably quite large. Assume that you have two hours of free time and you would like to
  watch a movie on Netflix. There are typically two ways you can arrive at the solution of which movie to watch. One is the most straightforward where you know exactly which movie you want to watch. This is a non-use-case with respect to data processing systems.
  The other, more interesting, use case is that you want to look at the recommendations of the movies and find the ones that have been most-watched as well as recommended by others. The reason is that the chances of you finding a good movie to watch increases significantly when you take recommendations into considerations. And the larger the recommendation set, the better the chances of finding a good movie. That is why movie-recommendation sites, such as IMDb, are so popular; they have the right set of tools or data processing algorithms in place to generate recommendations for their users. But, as you will realize when you design a data processing system, as the volume of the data increases, so does the cost associated with storing and processing this increased volume of data. With the ever-increasing volume of data, the key design decision for a data processing system is how to enable a large-scale processing of data while keeping the overall system cost low. This problem is only enhanced when you bring the variety and velocity of the data into the equation. The builders of the data processing system should carefully study and understand the behaviour of data within their environment to come up with the right set of tools for defining a data processing pipeline.
  To elaborate further on this statement, imagine you have a data collection system that collects data from a multitude of sensors that are deployed in the home-energy consumption-calculation devices (for example, Nest) being sold by the company.
  This data is collected to analyze the average use of a household and recommend ways to save on energy by looking at the general patterns of energy consumption.
  The volume of data could be potentially large if the solution is a hit in the market. Imagine this device being used in hundreds of households and the sensors sending data at a per-second rate.
  The variety of the data will not be that complex. In all cases, it will be in the range of a few different message types all following the same/similar data structure and pattern.
  The velocity of the data will potentially be high as the IoT devices will be sending data at a regular pace. Assuming no/minimal buffering is built into the collection systems, the data needs to be sent and processed by the data processing system at a high rate. This would mean more CPU-intensive systems and possibly parallel processing with multiple worker threads/nodes at each stage of processing.
  Thus, from this, you can determine that there needs to be a decoupled set of data processing logic that interacts with each other via some queuing mechanism and one class does not depend on the outcome of another class. In addition, things happen in an event fashion where each (batch of) dataset is treated separately from the others, and for each set, an event handler is always available. In such a case, the system can become
  distributed by nature as it is not possible to have commodity hardware and a monolith style of application.
  The other challenge of data processing systems lies in handling the other, and possibly more important, the cost associated with large-scale processing. This is the cost associated with latency.You may develop a data processing system that can perform the work of recommending the right hotel to you, but if the processing takes a week to generate a recommendation, then it’s not going to be useful to anyone.
  The tools that will be used for processing the data should be chosen carefully to allow for higher volumes of data to be processed without significantly compromising on the overall processing time.
  Another, usually ignored but very important, a challenge when understanding the right architecture and design of the data processing system stems from the traditional way of doing things and trying to fit them into your data processing design.
  Traditional ways of handling data no longer scale with the volume of data that needs to be processed in a data-intensive system.
  Traditionally, organizations bought huge computers with a huge price tag. These computers were provided by some top companies in the industry at that time. These systems worked and still work well for the kind of use cases they are supposed to handle. For example, if you know that the volume of your data will never increase and you have these systems lying around, then it still makes sense to use them in the data processing system you design. Unfortunately, the hypothesis that the data for processing will never grow generally does not hold, and there comes a time when these huge systems need to be complemented with more memory and more CPUs. This is generally referred to as scaling up the system. Scaling up the hardware is a very expensive process and, with the uncertainty on the volume of data that needs to be processed in low-latency, these costs can easily be measured in multitudes of dollars.
  Even if you invest a lot of money into these systems, there are still limitations as to how big a single machine/host can be. Thus, it becomes a challenge to shed the traditional way of approaching a system architecture and start embracing more modern ways of designing the systems. Shedding the notion of scaling up, and replacing it with the idea of scaling out, becomes more appreciative when the volume of the data increases.
  Unlike the scaling up architecture, where more and more horsepower is added to the same machine, scale-out architectures rely on adding more machines and distributing the workload across these machines in a consistent manner. Thus, if the data volume increases twofold, you simply need to add two more hosts instead of doubling the size of a single host. This applies in reverse situations as well. When the volume of data decreases, simply remove the hosts from the system and you can then use
  As you may have realized, scaling outcomes with its own set of challenges. For example, you need to clearly define strategies on how you would want to split the data into several independent chunks and then how to merge these results back again. Other challenges include developing a mechanism to know when these hosts are not responding and how to replace them with newer ones at a large scale in a very tight time-bound manner.
  Moving to the scale-out architecture for data processing requires significant engineering effort as well as mind change.
  If you found this article interesting, you can explore Anuj Kumar’s Architecting Data-Intensive Applications to architect and design data-intensive applications. This book is your gateway to build smart data-intensive systems by incorporating the core data -intensive architectural principles, patterns, and techniques directly into your application architecture.
  Anyone who has attended a fashion show can tell you there’s a great deal of waiting around: Models endure hours of hair and makeup. Attendees gather outside venues, and kiss cheeks. Hired cars idle. Champagne is passed. It’s these liminal moments—when many of us reach for our phones to pass the time—that interest the photographer Landon Nordeman.
  “That’s this place where photographs can exist, in this sort of unexpected, in-between space, where it’s not like this big grand movement or onstage moment,” says Nordeman. “It’s just this little thing that happens on the way to that. And that’s where you can get an interesting picture.”
  A consummate collector of killer details, Nordeman has photographed spectacles including the US Open, the Republican National Convention, and the Westminster Dog Show for publications ranging from Vanity Fair to National Geographic to the New York Times.
  But the world of fashion lends itself particularly well to his lens. Nordeman shows his appreciation with the blast of his flash, and captures weird, wonderful moments and details: A guest moved by the energy of a Marc Jacobs show drops to the floor and begins to do push-ups; a lineup of pink and caramel-colored eclairs shines on a platter backstage; a woman swipes moisturizer onto a male model’s chiseled torso.
  These moments, and many more, from the Fashion Weeks of New York, Paris, and Milan fill the pages of Nordeman’s forthcoming photography book, Out of Fashion, to be released on September 27.
  Here, in Nordeman’s own words, are the stories behind a handful of those photographs. (This interview has been edited and condensed.)
  This was backstage at Diane [von Furstenberg]’s show. All of a sudden Diane appeared from a little bit to my left, and came over to engage with André [Leon Talley], doing your sort of typical, “Hi, how are you,” kiss-kiss. I can’t remember what he was saying. I just remember that it was happening right in front of me. I’m holding the flash up with my arm, and I’m just shooting, shooting, shooting.
  The fashion crowd is a great subject because they like to be photographed. People at fashion shows know they’re going to be on display, and they’re expressing themselves with their fashion. When I’m photographing anyone who’s doing that, it’s my way of showing my appreciation. That part of it is a very honest and human reaction: When you see something you like, you want to photograph it.
  Those are the Beckerman twins. They’ve become a sort of known quantity on the fashion scene, Cailli and Sam. I didn’t know who they were. This was the very first time that I saw them, and I just went right up to them and said, “Can I take your picture?” I probably took a few, but this is the picture I knew I wanted. I just wanted the lips and the teeth and the mouth. It wasn’t about describing the fact that, “Oh these are two twins.” It was just seeing the similarity of the repetition of form.
  [The flash] heightens the intention of a photograph. If you saw this picture without the flash, the intention would have been not as clear, not as focused, not as exaggerated. By popping the flash in there, it really lets you know, “Okay we’re talking about the mouth, the smile, the red lips.” Everything is just nice and poppy, and that’s how I like it.
  That one is called Pink Pushup. That was at the Marc Jacobs show in New York. He turned the Armory into a total pink house, pink carpet, it was this entire pink space. I photographed before the show, people sort of silhouetted against the pink. I photographed the show itself. I had models walking on the pink floor, against the pink house. All of it was good. But this was the best moment.
  After the show, people were starting to clear out. And this woman started doing pushups on the floor. It just kind of happened out of nowhere! I saw her and started taking pictures. It turns out this woman is a fitness sort of specialist, and she said the reason she basically dropped to the ground and started doing pushups is she was just so excited and inspired by the show. In other words, it was her way of expressing appreciation.
  You could never plan for a moment like that. It’s just one of those things. When people spontaneously react, if you’re lucky enough to be there, you’ve got a great picture. There are pictures that you make, with your creative decisions: crouching down, the angle you choose, where you choose to stand, for example. This is just one that is more of a gift.
  Okay, I had like six of those. This was Paris Fashion Week. I think it was last March. What show was this? I think it was Vionnet. This was upstairs where hair and makeup was happening. It was this very old, small cramped backstage. I just kind of ended up standing next to these eclairs. And then I had one. Then I took a picture of it.
  I think in this instance I was responding to the color and the sort of graphic form, seeing the empty little packets, and the new ones. It’s kind of like, these eclairs were dressed up for the occasion in the same way that the models were being made up and dressed up, and it’s just sort of happened. But it’s not like I’m thinking all these things when I take the picture. I just react to what I see.
  That was New York [Fashion Week]. There’s a lot of skin being shown at Jeremy Scott—both guys and girls. If there’s any skin being shown, I’ve learned that it’s always going to get moisturized. I don’t know who made that decision. You see people putting moisturizer on women’s legs and arms. This guy was wearing—either he was wearing no shirt, or he was wearing a vest, and she was putting moisturizer on him.
  The person doing the moisturizing isn’t thinking about it. The model isn’t thinking about it. It’s got nothing to do with the clothes. It’s just this human interaction that happens, and if you look closely at it, it can sort of transcend the moment.
  It was Paris. It was last March, and this was leaving a show. I’ve seen Iris [Apfel] out, and she’s a very photogenic person. She’s used to being photographed, but I just had never gotten a photograph of her that I liked, until this one. She was being escorted out, and she’s in a wheelchair, which you can’t really see.
  As they were leaving this show I was walking backward and taking some pictures. And in this case, with Iris, I’m kind of like walking, talking: “Hi Iris! Oh, I’m working for New York Magazine!” She was talking to me. It’s just one of those things that like, you just kind of engage for a little bit. And then she went through a doorway, and that was it.
  Iris Van Herpen‘s clothes are always pushing the envelope of fabric and form. There’s a lot of synthetic clear fabrics that just don’t look like anything else. This is backstage. I think this is before going out onto the runway. It may have been coming back in.
  One photographer said, “If you want to take interesting pictures, stand in front of more interesting things.” That’s something I certainly appreciate and love about Fashion Week. This is something I just never could have imagined. I do love that it has this sort of faraway thoughtfulness to her expression. But I think this is about this fascinating design this person has created, and I’m just recording it.
  In New York, you can hear more of Nordeman’s stories at his book signing at Rizzoli on October 11, 2016.