text: US20150318020A1 - Interactive real-time video editor and recorder - Google Patents
  US20150318020A1
  US20150318020A1 US14/268,502 US201414268502A US2015318020A1 US 20150318020 A1 US20150318020 A1 US 20150318020A1 US 201414268502 A US201414268502 A US 201414268502A US 2015318020 A1 US2015318020 A1 US 2015318020A1
  US14/268,502
  2014-05-02 Application filed by Freshtake Media Inc filed Critical Freshtake Media Inc
  2014-05-02 Priority to US14/268,502 priority Critical patent/US20150318020A1/en
  2014-07-16 Assigned to FreshTake Media, Inc. reassignment FreshTake Media, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: PRIBULA, Alexis Juri
  2015-11-05 Publication of US20150318020A1 publication Critical patent/US20150318020A1/en
  The present application claims priority to and benefit from U.S. Provisional Patent Application No. 61/818,875, titled "Interactive On-the-fly Video Editing while Recording Video, and Related Social Video Components" and filed on May 2, 2013. The entire content of the provisional application is herein expressly incorporated by reference.
  In some cases, users may only know if video footage is worth keeping after it has been recorded. For example, this could happen at a soccer game where goals rarely occur but game action is frequent: many users would want to record all game action, but they might only want to keep the footage where goals were scored. This could also happen when trying to record the first time a baby walks: it might take many tries before the baby actually walks, but people generally record each attempt just in case. Therefore, in some embodiments, the video management system may be used in conjunction with a sliding window of video and metadata; it would only keep the video-montage and associated metadata for the last 5 shots or if the shots started during the last 3 minutes (or any other predetermined duration rule). Anything older could automatically be discarded, unless the user indicated that they wanted to keep it. This means that the user would have up to 3 minutes to decide if what is being recorded is worth keeping. For example, when recording game action at a soccer game, a user could press a button 2 minutes into the action after they see a goal has been scored in order to indicate they intend to keep the video montage of the action leading to the goal. Everything else recorded would automatically be discarded. This might be a particularly interesting interaction for users wearing glasses capable of recording video. Indeed, with glasses, a user would not need to pull an electronic device out of her pocket or purse to start recording video, the glasses would always be readily operable, keeping the last few video shots created with the video management system available just in case anything interesting happened. This could also work with any electronic video recording device with continuous recording capability: head/hat/caps/beany mounted video cameras, hidden clothes cameras, eye lenses capable of recording or transmitting video wirelessly, glasses capable of transmitting video (with a cable or wirelessly via Bluetooth/Wifi/network/etc.), smartphones, smart-tablets, photo cameras capable of recording video, any video recording device held in position to record video such as professional broadcast cameras, etc.
  Additional examples of the types of feedback the user might be getting while recording video with the video management system include traditional cinematographic recommendations possibly using continuous animation and interactive feedback UIs (such as but not limited to video preview color filters), in particular if the user is zooming in by rotating the electronic device as described below, the video management system may encourage smooth zooming by fading from color to gray-scale; if the zooming is too abrupt, it may discourage users from recording video when lighting is too low by first flashing a warning icon/message and then possibly starting to discard video; if the user is doing a head shot or an over the shoulder shot, it may detect faces using face detection and super impose ideal subject placement sketches on the video preview to encourage better framing, if possible analyze the position of the head(s) and other items in 3D, and recommend the ideal positioning of the recorder in a 3D space to improve framing, e.g. "for a better head shot: take one step back and half a step right". The video management system may also detect conversation or action by detecting people/faces/speech/action and encourage the 180° cinematographic rule by analyzing electronic device motion and overlaying 3D sketches of an 180° wall or line, detect objects, subjects, architecture, surrounding nature and encourage the ⅓-⅔ cinematographic rule or otherwise encourage a symmetric video image composition by overlaying dynamic guidelines on the video feed preview. It may emphasize moving objects inside the video feed preview by surrounding them with fictive UI frames and encourage the recorder to smoothly track their movement by flashing the fictive frames red, fading the video feed preview screen to gray-scale. It may display warnings if the fictive frame moves out of the actual video frame), warn user if background noise is so loud that the video management system will not be able to automatically remove it from the soundtrack, identify subjects talking in the video by surrounding them with fictive frames to inform the recorder that someone talking has been detected (and that they will likely be less interrupted by the video management system in the on-the-fly video editing), identify expressions of emotion in the video with fictive frames (to inform user they will likely be less interrupted by the video management system in the on-the-fly video editing), and display any type collaboration feedback to the UI for teams recording video together. More specifically, when creating the special video effect of freezing time and rotating the camera around the action, the video management system may provide live feedback to inform all video recording group members if they are well positioned and pointing in the best direction so that each one of their video image frames can used to recreate the time freeze rotation of the camera around action or point of interest. The video management system may, if music has been identified as playing in the video feed soundtrack, display a visual indication that this is the case by displaying the title and/or artist of the song and/or a video-preview color filter, display suggestions of content that could be recorded with video (if the user is close to the Eifel tower, suggest filming it, if there is a video meme that is currently popular, suggest performing it, if the meme is specific to a type of location which happens to be nearby, suggest going there and performing the meme, etc.)
  The video management system could encourage users to record using video templates. In particular, these video templates could potentially be based off of video montages created with the video management system. One example of a video template is a Harlem Shake meme video based off of a Harlem Shake video template featuring a first 15 second video shot of a person dancing alone next to an unfazed group/crowd, a second video shot of the same group/crowd dancing including the person dancing alone on the first shot, and the song "Harlem Shake" by artist "Baauer" as the soundtrack for the whole video montage and starting at same time as the start of the montage. A second example of a video template is one for a tourist trip in Paris where the user would be encouraged to record shots of various landmarks (such as the Eiffel Tower, the Arc de Triomphe, etc.) and to add commentary to these shots (see below for an example of using this with video shot links). A third example of a video template is one for the first few steps of a baby featuring a video shot of the baby's excitement at seeing other babies walk (around 5 seconds long), video shots of a few failed attempts (around 10 seconds long in total), video shots of the baby's first successful steps (around 7 seconds long), and a few final video shots of big smiles from everyone (around 6 seconds), with the song "What a Wonderful World" by artist "Louis Armstrong". A fourth example of a video template is one where a group of friends want to create a video montage for someone's birthday, they could send out a video shot to all friends saying "Say happy birthday to John, it's his birthday on Tuesday, let's all surprise him with a great video!" followed by an example happy birthday video shot, all resulting happy birthday shots would be stitched together into a video montage and then sent to John on his birthday. These are just examples, video templates could work for the creation of any type of video, and could use any song or sound in the background. Meme based videos could have the extra added benefit of having a higher chance of becoming viral themselves, which could help the virality of the product through in-video messaging.
  In some embodiments, users could vote on templates, and templates could be continuously ranked and suggested to the users of any video management system based product. For example, video templates could be suggested to the user through notifications (by the operating system of the electronic device used, by the browser of the electronic device used, by an app running on the electronic device used, by email, etc.), they could be stored in a section of the video management system based app or website, they could also be suggested when users start the video management system video recording interface, etc. In particular, when starting the recording interface, there could be an inspiration screen displaying suggestions of items to record (such as but not limited to: very short text, icons, small designs, video screenshots, etc.), and these suggestions could be linked to a video template that could help and guide the user in creating better looking video montages. There could also be shot specific suggestions displayed just before recording a specific shot.
  The smoothed ADA values computed in Step 620 can also be used to provide feedback on how truly stable the electronic device is while recording a shot. If the ADA values are much higher than the threshold value, then live feedback could be given to the user to indicate that the device is completely truly stable (such as but not limited to a UI on the electronic device displaying a preview of the video feed in full bright color). If the ADA values get closer to the threshold, which might cause the recorder to trigger a shake- to-cut gesture, then live feedback could be given to indicate this state (such as but not limited to a UI on the electronic device displaying the video feed in gray-scale, in particular, the ADA values could be mapped directly to a parameter controlling the amount of color in the video preview; a dimmed gray-scale video preview could be kept for when shots have actually been cut and video is being discarded).
  In some embodiments, the analysis of movement can also be complemented with more traditional interfaces such as the press of a button (on screen or not). For example, a user could press a button while recording footage and the video management system would try to find a stable region of video starting before and ending after the moment when the button was pressed. The resulting shot would actually start a few seconds before the button was pressed when the video feed started stabilizing and end a few seconds later when the video feed starting shaking. If storage space is an issue, a sliding window video buffer could be used, which would only keep the last 20 seconds of video just in case the user presses the button. This interface might be better adapted in cases where the button is not pressed on the same device as the one recording video. Indeed, if the user pressed a button on the same device as the one recoding video, it might cause an involuntary shake in the middle of the recorded video shot, which might be hard to smooth out algorithmically later.
  A second example that uses these links are video templates where the template author suggests certain items with or without the video management system, as discussed above. For example, a template could be a video montage where each video shot provides instructions on what to record. This could work for the example of the Paris tourist video template discussed earlier: the template could be a video montage with video shots of the popular landmarks (such as the Eiffel Tower or the Arc de Triomphe) with extra information and recording recommendations in the voice commentary. The video management system recording interface could play each landmark template shot just before letting the user begin recording. This would make sure that the user(s) has all the relevant information and instructions to record the next few shots just before he/she/they started recording them. When a user recorded a shot as indicated, the shot in the video montage template could link to the newly recorded video shot. More generally, the template shot could then link to all the recorded "answer" shots, and users could vote on the best answers. Votes could be used to rank the shots best answering specific video template shots or rank the video montages best answering a video template (as mentioned briefly above with votes). Many other possible video template link examples are possible.
  1. A method performed by an electronic device operated by a video shooter of creating a video montage, comprising:
  recording video footage;
  generating feedback on the recorded video footage before or while additional video footage is recorded;
  capturing sensor data while the video footage is recorded;
  detecting, from the captured sensor data, a first signal associated with beginning a video segment;
  detecting, from the captured sensor data, a second signal associated with ending a video segment;
  extracting the video footage recorded between the first signal and the second signal into a video segment;
  determining whether to retain the video segment based on one or more of the generated feedback and the captured sensor data; and
  creating a video montage comprising retained video segments.
  playing a music segment,
  wherein recording video footage includes synchronizing playing the music segment and capturing images.
  3. The method of claim 1, wherein the electronic device is a laptop, a desktop, a cellular phone, a wearable device, a tablet, a television, or a video camera.
  4. The method of claim 1, wherein generating feedback comprises:
  determining a quality of the recorded video footage by analyzing the recorded video footage; and
  showing one or more of information on the determined quality and a recommendation on recording video footage based on the determined quality.
  5. The method of claim 1, wherein generating feedback includes gathering feedback on the recorded video footage provided by a viewer of the recorded video footage.
  transmitting the created video montage to a social cloud platform for storage; and
  receiving the feedback provided by the viewer from the social cloud platform or another electronic device.
  7. The method of claim 1, wherein the sensor data includes data related to a speed, a course of movement, a temperature, a pressure, a humidity, a location, an acceleration, or an orientation of the electronic device.
  8. The method of claim 1, wherein the sensor data includes data related to a speed, a course of movement, a temperature, a location, an acceleration, an orientation, a facial expression, an utterance, or a gesture of the video shooter.
  9. The method of claim 1, wherein the first signal results from holding the electronic device steady for a predetermined period of time, holding the electronic device in a predetermined orientation, operating a predetermined piece of the electronic device, making a predetermined gesture, or uttering a predetermined word.
  10. The method of claim 1, wherein the second signal results from putting the electronic device down, shaking the electronic device horizontally, holding the electronic device in a predetermined orientation, operating a predetermined piece of the electronic device, making a predetermine gesture, or uttering a predetermined word.
  11. A method performed by a video management system of creating and using a video template, comprising:
  receiving footage of a first video montage from a first electronic device operated by a first video shooter as the footage is being recorded;
  generating a video template of metadata associated with one or more shots of the first video montage while or after the first video montage is being created;
  transmitting the received first video montage and the generated video template to a social cloud platform for storage;
  receiving a request to access the video template stored on the social cloud platform from a second electronic device;
  transmitting the stored video template from the social cloud platform to a third electronic device operated by a second video shooter in response to the request;
  receiving footage of a second video montage that is created based on the transmitted video template from the third electronic device as the footage is being recorded;
  generating a recommendation on creating the second video montage based on the transmitted video template and the footage received from the third electronic device; and
  transmitting the generated recommendation to the second or the third electronic device.
  12. The method of claim 11, wherein each of the first, second and third electronic devices is a laptop, a desktop, a cellular phone, a wearable device, a tablet, a television, or a video camera.
  13. The method of claim 11, wherein the metadata includes information regarding duration, image content, background music, dubbing, or subtitling of one of the shots or the first video montage, information on an editing effect applied to one of the shots or the first video montage, or information on a transition between two of the shots.
  14. The method of claim 11, wherein the recommendation includes information on conforming footage of the second video montage to the video template or information on applying editing effects specified in the video template to the footage.
  15. The method of claim 11, further comprising receiving feedback on the video template from the second electronic device.
  receiving a search query for a video template stored on the social cloud platform from the second electronic device; and
  transmitting a list of stored video templates that are hits of the search query to the second electronic device.
  17. A method performed by a system running on a social cloud platform of creating and using a video template, comprising:
  storing the received first video montage and the generated video template;
  receiving a request for the stored video template from a second electronic device;
  transmitting the stored video template to a third electronic device operated by a second video shooter in response to the request;
  18. A method performed by an electronic device of creating a video montage from a video template, comprising:
  transmitting a request to a social cloud platform for a video template of metadata associated with one or more shots;
  receiving the video template from the social cloud platform in response to the request;
  generating footage of a second video montage;
  generating a recommendation on creating the second video montage based on the received video template and the generated footage; and
  generating additional footage based on the generated recommendation.
  wherein the video template is created from a first video montage,
  further comprising receiving the first video montage from the social cloud platform in response to the request,
  wherein generating footage of a second video montage includes recording new footage or reusing footage of the first video montage.
  20. The method of claim 19, wherein reusing footage of the first video montage includes reordering the shots in the first video montage, adding shots to the first video montage, removing shots from the first video montage, applying an effect to a shot of the first video montage, or reusing some of all of a shot of the first video montage.
  21. The method of claim 18, further comprising transmitting feedback on the video template to the social cloud platform.
  22. The method of claim 18, further comprising showing the generated recommendation on a user interface of the electronic device.
  transmitting a request to view the list of stored video templates to the social cloud platform; and
  receiving the list of stored video templates from the social cloud platform.
  US14/268,502 2014-05-02 2014-05-02 Interactive real-time video editor and recorder Abandoned US20150318020A1 (en)
  US15/237,908 Continuation US9870798B2 (en) 2013-05-02 2016-08-16 Interactive real-time video editor and recorder
  US20150318020A1 true US20150318020A1 (en) 2015-11-05
  US20150341547A1 (en) * 2014-05-21 2015-11-26 Motorola Mobility Llc Enhanced image capture
  US20150341549A1 (en) * 2014-05-21 2015-11-26 Motorola Mobility Llc Enhanced image capture
  US20160118085A1 (en) * 2014-10-28 2016-04-28 Obshestvo S Ogranichennoy Otvetstvennostyu "Sinezis" Method and system for systematization of production-process video data
  US20160227095A1 (en) * 2013-09-12 2016-08-04 Hitachi Maxell, Ltd. Video recording device and camera function control program
  US10338407B2 (en) 2017-06-26 2019-07-02 International Business Machines Corporation Dynamic contextual video capture
  US10362340B2 (en) * 2017-04-06 2019-07-23 Burst, Inc. Techniques for creation of auto-montages for media content
  JP4468789B2 (en) 2004-11-08 2010-05-26 日本電信電話株式会社 Bml content providing system
  US9774776B2 (en) * 2013-09-12 2017-09-26 Hitachi Maxell, Ltd. Video recording device and camera function control program
  US9536570B2 (en) * 2014-10-28 2017-01-03 Camera Bi Limited Liability Company Method and system for systematization of production-process video data
  US9858966B2 (en) * 2015-07-28 2018-01-02 At&T Intellectual Property I, L.P. Digital video recorder options for editing content
  US10347292B2 (en) * 2015-07-28 2019-07-09 At&T Intellectual Property I, L.P. Digital video recorder options for editing content
  US10049702B2 (en) * 2015-07-28 2018-08-14 At&T Intellectual Property I, L.P. Digital video recorder options for editing content
  US20170186464A1 (en) * 2015-07-28 2017-06-29 At&T Intellectual Property I, L.P. Digital Video Recorder Options For Editing Content
  US20160358629A1 (en) 2016-12-08
  Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PRIBULA, ALEXIS JURI;REEL/FRAME:033328/0776