Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér.
When we were children, every single one of
us dreamed about having a magic pencil that
would make our adorable little drawings come
true. With the power of machine learning,
the authors of this paper just made our dreams
come true.
Here's the workflow: we provide a crude drawing
of something, and the algorithm fetches a
photograph from a database that depicts something
similar to it. It's not synthesizing new images
from scratch from a written description like
one of the previous works, it fetches an already
existing image from a database.
The learning happens by showing a deep convolutional
neural network pairs of photographs and sketches.
If you are not familiar with these networks,
we have some links for you in the video description box!
It is also important to note that this piece
of work does not showcase a new learning technique,
it is using existing techniques on a newly
created database that the authors kindly provided
free of charge to encourage future research
in this area.
What we need to teach these networks is the
relation of a photograph and a sketch.
For instance, in an earlier work by the name
Siamese networks, the photo and the sketch
would be fed to two convolutional neural networks
with the additional information whether this
pair is considered similar or dissimilar.
This idea of Siamese networks was initially
applied to signature verification more than
20 years ago.
Later, Triplet networks were used provide
the relation of multiple pairs, like "this
sketch is closer to this photo than this other
one". There is one more technique referred
to in the paper that they used, which is quite
a delightful read, make sure to have a look!
We need lots and lots of these pairs so the
learning algorithm can learn what it means
that a sketch is similar to a photo, and as
a result, fetch meaningful images for us.
So, if we train these networks on this new
database, this magic pencil dream of ours
can come true.
What's even better, anyone can try it online!
This is going to be a very rigorous and scholarly scientific experiment - I don't know what
this should be, but I hope the algorithm does.
Well, that kinda makes sense. Thanks, algorithm!
For those Fellow Scholars out there who are
endowed with better drawing skills than I
am, well, basically all of you - if you have
tried it and got some amazing, or maybe not
so amazing results, please post them in the
comments section! Or, as we now have our very
own subreddit, make sure to drop by and post
some of your results there so we can marvel
at them, or have a good laugh at possible
failure cases.
I am looking forward to meeting you Fellow
Scholars at the subreddit. Flairs are also available.
Thanks for watching, and for your generous
support, and I'll see you next time!
