- Stephen Bury, and I am the
Andrew W. Mellon Chief Librarian
of the Frick Art Reference Library.
I'd like to welcome you to the third
Digital Art History Lab Lecture.
Today we are very pleased to have
as our lecturer Rick Johnson.
In 1977, Rick received a PhD in
electrical engineering
from Stanford University
along with the first PhD minor in
Art History granted by Stanford.
Following four years on the
faculty at Virginia Tech,
he joined Cornell University in 1981
where he is the Geoffrey S.M. Hedrick
Senior Professor of Engineering,
and also the Jacobs
Fellow in Computational
Art History at Cornell Tech.
In 2007, in collaboration
with the Van Gogh Museum,
he founded the Thread Count
Automation Project, T-CAP.
He has also worked on other computational
art history projects supported by
the CREST Foundation,
Museum of Modern Art,
the Rijksmuseum in Amsterdam,
and RKD in The Hague.
After the presentation,
the Frick Collection's
Association Research
Curator Margaret Iacono,
a specialist in northern
European paintings
from the 15th to the 17th centuries,
will hold a conversation with Rick
discussing the impacts of his work
on the study of various paintings
including Mistress and Maid, the focus
of an exhibition that
Margaret is preparing.
This will then be
followed by a Q&A session.
Please welcome Rick Johnson.
(applause)
- You shouldn't clap
until you hear if I have
something interesting to say right?
(laughter)
So thank you for coming.
This has been an exciting decade for me.
Stephen just mentioned that I started
working on thread counting in 2007.
How many people know what that means?
Oh goody, but not everybody.
So that's where we'll start.
I'll tell you how and why a little bit,
and I'll weave in a little bit about
the story of how this happened.
Essentially what you're seeing is probably
one of the good things
that comes out of tenure.
So about 10 years ago I decided that
they couldn't fire me if I decided
not to do engineering for a while.
But in fact it happened
right about the time
that digital humanities was taking off.
So what I felt like at
the beginning was that
I could use image processing to help with
things like attribution
because as I remember
from my art history courses last century,
that was a large part of what we did
was you're trying to
identify the painting,
date the painting, figure out
who the artist is, et cetera.
And there was always the idea that maybe
you could do that with a little bit
of machine learning,
everybody knows what that is
because it's in the paper all the time.
It turned out that that was
a whole lot harder than I'd imagined.
And also at that time I was looking
for problems that, since
the field didn't even
have a name yet, now we call it
Computational Art History, who were using
the computer to help answer
art history questions.
Then what I wanted to
do for the first time
in my career actually
was do original research
that I could involve undergraduates with.
We talk about that a lot
at research universities,
the research experience for undergraduates
and trying to get students excited
about doing new things,
but usually that means,
it's like when my father would try to
show me how to work on the car, I got to
hold the flashlight because I couldn't
be trusted to do anything else.
So the graduate students tend
to have to just watch to a great extent
because the background
they need is too dense.
So what we're actually gonna see today
is a technique that anybody as an
undergraduate engineer could've done.
So the mathematics and the
technical part is fairly modest.
And it's immediately
turned into something,
or very quickly turned into something
that was way beyond our imagination
as to how useful it might be in
answering art history questions.
And that's not to say that we
thought of thread counting.
Let's look at this x-ray for a minute.
This is a positive, not the negative,
but it's a two centimeter square.
So what I'm gonna teach you
first is how to count threads.
With this image I think you can see,
this looks a little bit
like a piece of fabric.
It's got some metal thing in the way
that hurts the x-ray, and I'll talk about
why an x-ray in a minute, but essentially
the idea is you put on a headset,
that has a magnifying ability, and it has
some size to it that's known, each side
here is two centimeters, so you
would start counting each thread.
However many there are, say 40 or so,
and you divide by two centimeters and that
tells you the number of
threads per centimeter.
Got it?
You're supposed to say yes.
- [Audience] Yes.
(laughter)
- Okay.
And you do it in this
direction and this direction.
Okay, now.
The question is why would you do that?
And I didn't know, somebody had
to point out this problem to me.
So I spent about a year and a half
visiting the Van Gogh
Museum starting in 2006.
They were the first
museum that I approached
just because I had a friend that
was already interacting with them.
So I followed the head conservator
around for several months.
I would visit every two months
or so for about 10 days, and she would
show me this problem and that problem
and I would try to figure out is there
some way I can help by using the computer.
So after about a year and a half of that
one day she says we're
gonna count threads.
I didn't even know what that meant.
I mean I'm not the one in my family
that buys the sheets or the pillow cases
where the thread count matters right?
Okay, but the idea was that if you had
two pieces of canvas that originally
came from the same roll, the idea
would be is that they would have to have
basically the same average thread count.
Pretty reasonable right?
If they didn't have the
same average thread count,
they couldn't be from the same roll.
Unfortunately, even if they did have
the same average thread count they
might not be from the same roll,
so it was kinda a negative forensic,
but it was used quite extensively
starting in the 70s and 80s by
the Rembrandt Research Project.
But there was a big success
in Van Gogh research.
Around 2000 there was an exhibition
at the Art Institute in Chicago where
they were able actually to find all
the paintings that they had done on a roll
of jute that was mentioned
in Van Gogh's letters.
And 20 meter rolls and they took all
the paintings they thought were made
during that period by either Gauguin
or Van Gogh, and then they did
the tread counts and they found
all the ones that had a very low
thread count for the course of jute,
and when they laid them all out it was
20 meters and it turns out that a couple
of paintings in there
had been doubted and that
led them to claim that
they were actual Van Goghs.
This is why the Van Gogh
Museum was interested.
Could we do that?
I get to show you some nice pictures,
so here's a very nice, all Vermeers okay?
Why Vermeer, Van Gogh is a great...
If you will, was a great place to start.
Why?
Because the Van Gogh Museum has
such a large number of his paintings.
And the Kroller-Muller Museum in
the Netherlands has the
second largest collection.
It was a way to actually, and we also know
from his letters that
about 60% of his canvas
was bought by his brother by the roll.
That project has advanced to the point now
where we've put together
about, we're trying
to put together about 40 or 45 rolls
of canvas which allows
us to date paintings
to within about a three week period
because he went through
the canvas so fast.
So it turns out that that dating mechanism
is more accurate than any other mechanism
that's tended to be used
for Van Gogh paintings.
But the real reason I got into all of this
was because of my love for
17th century Dutch art.
I have to admit Rembrandt
was, like a lot of people,
was the first artist that
somehow emotionally drew me in.
So after we were doing the Van Goghs,
(mumbles) the conservator at the
Kunsthistorisches Museum
in Vienna asked us
to do one of those weave
maps for her Vermeer.
And it worked.
So we thought, my
reaction, typical American,
let's do everything, I
thought oh, let's do them all.
There's only 34.
I mean how hard can that be?
But it took us four years to collect
the x-rays from the different museums.
About five years actually with the help
of Walter Liedtke formerly at the Met
who has been, was just an absolute
terrific mentor for me
as I got into this area.
So the why is in order to see if you can
come up with a way to tell if two pieces
of canvas were originally
on the same roll.
What would that tell you?
Well if you knew other things, none of the
forensics that you come by technically
are an answer by
themselves of the problem.
But going with the other things we know,
for instance Van Gogh
was a solitary figure,
except for that time with Gauguin,
he rarely shared canvas.
If you showed that the
two pieces of canvas
were at one point
connected together and you
can look in his letters, you can probably
put a case together for saying whether
or not those were both done by Van Gogh.
So the why is to give you an extra piece
of information about trying to say
the two canvases at one point were
at least geographically located
in the same place, and with other
information that should be able
to answer questions that you might have
about authenticity, dating, and some
other things which we'll get into
when I show you some of the results.
And how, I already mentioned to you,
you have this elaborate
headset and you actually
count the threads, and
then what do you do?
Well this is, it would be an actual film.
So it's on a light box.
And you can't mark it because it's
a valuable piece of information.
Which meant that you
couldn't record where you
did the count, which meant
it wasn't repeatable,
which means it's scientifically useless.
In the sense that you
couldn't validate it.
So the thread counting
also, how many spots
do you need to get a good sense
of the average across the whole canvas?
Well it turns out you need more
than you probably wanna do.
So what happened was is it would be
something that you didn't do to the extent
where you counted in every place
across a painting 'cause one painting
might take you a couple years to do
that type of thing, especially if you
take lots of coffee breaks, all right?
So it looked like a problem that
would be good for engineering.
Why?
Because if you look at this pattern
that goes light and then dark
and then light and then dark,
if you wanna think of the light color
as being a high number, and the dark color
as being a low number,
what you end up with
is some signal that goes like this.
So it's a periodic signal,
just like your heartbeat.
It turns out that's one
of the problems that
engineers have solved a
thousand different ways.
There's so many methods,
so many algorithms
for actually determining the period
of these fluctuating signals.
So it turns out the
standard thing is called
a Fourier transform, don't worry about it.
It's named after the French mathematician
that invented it, but it's something
we teach every sophomore
electrical engineer.
So I thought this is perfect.
This looks like a great problem.
So I said to Ella Hendriks who was
the conservator of the Van Gogh Museum
can you scan this for me 'cause nothing
was digitized 10 years ago when I started.
And I took it back to my apartment
in Amsterdam and wrote
up a little program.
Sure enough my hand count matched what
the algorithm gave me, so the next day
I went back to the museum and I said
we'll count all the
paintings in the museum.
And the Dutch culture
is a little different
from ours in the US, and
so their reaction was
don't be so American, so eager
to overdo everything, take it easy.
We'll give you x-rays, scan some x-rays,
30 paintings or so, and
take it easy, don't worry.
But it turned out we've now done
more than 500 Van Gogh paintings.
We've done all the ones in the
Van Gogh Museum and the
Kroller-Muller Museum.
And as I said earlier,
we found heaps of matches
because we know he bought
canvas by the roll.
So we had this, I mean I
love 17th century Dutch art,
so I had this possibility
of doing Vermeer's.
And experts that I talked to said
this sounds like a
great idea, but you know
you're not gonna find any matches.
Vermeer, he only painted two a year.
What would he be doing, does he
buy canvas and then horde it for a while?
We don't know 'cause
we have no information,
no documentation about
his studio practice.
But there was a paper written when the
only privately-owned
Vermeer had its last sale.
It's the one here at the Leiden
Collection in New York City.
There was a statement
in a paper that I read
that said oh, the canvas
in that is exactly
like the canvas in The Lacemaker.
Exactly is a scary word to use 'cause it's
often not true, but close enough,
and I thought oh, let's look at
those two, which we will later.
So the motivation was there.
There were people that felt like some
of the canvases looked
like they were the same.
So the how, I told you about the headset.
The counting of all these spots,
and that's what the reel is for because
it's just an unbelievably tedious task
and that's exactly what I was looking for.
The computer doesn't sleep, it doesn't
need to take a coffee break, okay?
So it doesn't matter how tedious it is,
in fact that's what it should be used for.
So the idea was to say that we could take
every two centimeters square, like this,
and do the counting across the
entire surface of the painting.
You're supposed to say so what.
(laughter)
So what?
Well now what I've got
is ability to average
the counts across the entire painting,
so at least I'm gonna have a statistically
significant average thread count.
That's as far as we
thought it through because
all we'd been asked to
do was count threads.
They didn't ask us to do anymore,
but let's think about
it just a little bit.
What do you expect to see?
Sorry, I realize I'm starting
to sound like a professor.
Trying to make you think
during the lecture,
but hang in there with me okay?
How many people have done weaving?
You're kidding me.
Oh okay, a few hands.
But at least you know
what a loom looks like
where you've put the threads on the loom.
They're under tension as they go through,
or they're on one roll,
they go through everything,
and they come out when
you're making the fabric.
The tension means when
you set up the loom,
which is a pretty complicated process,
that the threads at this
part are going to have
the same thread count,
or the same density,
as you move through the fabric.
In this direction.
But the bunch just to the left of it
might be a little different, here you can
see that easily because
you can see that some
are way thinner than some
of the other threads.
So what we decided to do was, because you
can't give art historians a bunch
of numbers and graphs right?
It doesn't work.
So what do art historians
do that engineers don't?
They know how to do close looking.
So they're very used to looking at things,
especially things that look like
paintings or art objects, and deciding,
making observations and insight,
is drawn from that close observation.
That's why we decided okay, what we're
gonna do is we're going to, instead of
giving you a number that goes in every
two centimeter square they say,
I'm gonna put a color there.
So I'm gonna put a color so that
the thread counts that
are high have red colors
for instance, and the
thread count that are
low have blue colors
or something like that.
So now that I've told you that I'm gonna
count everywhere, and that the count's
in a column for the
threads going this way,
should be similar in
count compared to the ones
over here, they'll also
be similar to each other,
but different from this column here.
Same thing for rows.
So what would you expect
that picture would look like?
Well we didn't know either, we didn't even
think about it, we just drew it.
This is the problem is that when you think
something's impossible
you don't let your brain
go there and imagine what you might see.
So immediately what we saw.
Oops, sorry, wrong one.
What we saw was there were stripes.
I'm getting ahead of the story here,
but the point is these stripes are
the same color because the threads have
the same density all the way through.
Turns out that the stripe pattern's
different for every roll of canvas.
Then you have a fingerprint or a bar code
that allows you to identify the canvases
as being from the same roll.
That's where we're headed.
But in order to narrow down the ones
you want to compare to each other,
the first thing we'll do is we'll take all
the average values of all
the different paintings
and see which ones are
close to each other.
It took us five years, but
we have all the Vermeers.
Here's the, it's what's
called a scatter plot,
but essentially this plot that shows you
the thread count, so
the smaller thread count
might be 12 1/2 threads per centimeter,
and the other one is a little bit larger,
so that would be painting number 36.
There's no reason for you to know
what painting number 36 is.
It turns out it's the one owned by
the Leiden Collection, but these are
numbers from Walter Liedtke's catalog.
Rather than trying to write the name
of each one next to it
I just put a number,
and if you wanna ask me which one's
which later we can do that.
So the thing we know
is that what we want to
look for are those that
are close to each other.
Because only the ones that are
close to each other have a chance.
That's the first test, the average
thread count needs to be the same.
Or very close.
How close?
So what you do is you go read some things
and it turns out the
Rembrandt Research Project
decided that if it was more than
one thread per centimeter different,
forget it, it can't be the same roll.
So what I'm gonna do is I'm gonna draw
a box around each of these locations
that's two centimeters on a side.
So I'm gonna capture everything
that's only one thread per centimeter
away from the one I'm interested in.
So you can see that number one
doesn't have any count matches,
so there's no reason to check
its weave pattern against anything else,
there's no chance that it's gonna match.
Whereas some of the others clearly have
a large number that
are close to each other
because one of the things about Vermeer
is that there's a huge
number of these paintings
that have about 14 threads per centimeter
by 14 threads per centimeter.
And this was something
that was already known,
but if you count them crudely you, again,
all this tells you is they might be
from the same canvas roll, but not
with confidence that they are.
So what we're gonna do is we're gonna
try to find all the ones that are
close to each other and compare them.
There's the stripes.
So we're gonna take, for
instance, this painting
at the top which is L18,
don't worry about that.
But see the stripes?
These are the thread
counts in every little,
it turns out, a half centimeter square.
And then we put a color in it.
I tried to use the oranges
and purple so you can
use lots of different colors
just to make them vivid.
But what we want to do is compare
the top row, so this is the horizontal
thread count 'cause
you can see the threads
going horizontally and see the stripes,
and this is the vertical thread count
of the painting that's of interest to us.
And it turns out that if you go back
to L18, it's up there at the top,
these two, 14 and 19, are very close.
So those are the ones we're gonna check.
This is the weave map for 14.
This is the horizontal threads in 14.
With the average against which these
colors are drawn matching
the average in this
painting so that they're
on the same color scale.
This is the vertical threads for 14,
but to the average scale
for this painting here,
for this version here, the horizontal.
Notice it's really dark
because look, the horizontal
thread count is 21, the vertical's 15.
So you know there's no
way you're gonna match
the vertical of this one to the horizontal
of that one, but in other cases you might.
You don't know whether they turned
the canvas before they used it or not.
So we'll do the horizontal and vertical
to the same horizontal average, and then
we'll try to see if either
of these match that.
Then we'll do the same thing here.
This is the vertical, this is the vertical
for 14, and this is the horizontal for 14,
turned 90 degrees, and of
course is almost all white
which means it's very far
from the right colors.
And when they're gonna do it again for 19.
With me so far?
Nod your head.
Okay, do you know what
a bobblehead doll is?
Okay, where I grew up you put them
on the dash of your pickup truck right?
It's usually your
favorite football team and
the head bobs when you drive
around in the countryside.
This is a technique I teach my students
so I'll know they're still awake
while I'm giving a lecture.
If I ask you a question, you don't
have to say yes or no, you just
move your head a little bit okay?
So I'll know everything's going all right.
So let me try that again.
You got it?
We're gonna compare this
one to this one to this one,
and this one to this one to this one.
These are clearly out of the question.
With me?
Okay great.
Okay, so let's try this one because wow,
look at this, this is really,
it looks very similar.
If I move this one over just a little bit
notice all this dark stuff lines up.
You know, maybe.
And then over here, this
one, this is not easy
by the way because it takes close looking,
which art historians can do, but if I take
this one and move it up on the side
over here of this one,
see this light stripe?
It matches up with this light stripe.
Turns out that these are two of the pairs
that nobody expected to find.
So let me show you what happens
once we figure out which ones match.
Turns out we have a
group of four that match.
By now positioning them properly,
you can see there's the dark stripe
between these, so that's 14, that's 18,
this is 19, and this is 31, okay?
That's what they say here.
But here's the picture of paintings.
Here you can see the
thread count this way.
If they're aligned this
way they have to match.
And they have to match this way
if they're aligned horizontally,
so these four pieces came from
the same original roll of canvas.
Now at this point you're
also supposed to say so what.
So if we knew a lot more about Vermeer,
which we don't, we might say, in fact
many experts do, and this
is the part I don't do.
I just show this to the art historian
and say well do you think these match?
What do you think it means?
And then I'm done.
Because I don't have that expertise.
Notice I'm not trying to
push them out of the...
I mean all I'm doing is giving them
an extra tool to use with their analysis.
It's not trying to supplant
the art historians,
which is I think the way a lot
of machine learning is set up.
We won't need you humans anymore.
The machine will take care of it.
But that's not the case here, I'm thinking
this is just the tool that actually
extends the reach of the art historian.
So it turns out that this would be
totally unexpected that you would
find four pieces of canvas why?
Because these three are dated about
six years different from that one.
Now either this means
the guy is the world's
biggest pack rat, or
we got the dates wrong.
Right?
So now you need more information to answer
that question, but it puts you on a path
that wasn't there before in terms
of trying to understand which ones
he did together in which time.
This is very important to art historians
because what you really wanna know
is how the sequence of pieces of art
were developed so that you can follow
the style changes and the intent
of the artist over their career.
So if you get them out of
order, you're in trouble.
You're gonna make a lot
of silly conclusions.
This was our most recent one, the others
I'm gonna show you, the other three pairs,
so there's six pairs we found
out of the 34 paintings, how about that?
It turns out there's 10
paintings that match another one.
That's a large fraction of his paintings,
which means that tells us something.
We don't know exactly
what, but it definitely
tells us something about
his studio practice,
how he acquired canvas,
and so on and so forth.
Does everybody believe me so far?
Does that look like a match to you?
I'd love to walk into the audience,
but if I do that I'll kill myself.
But basically what we
do is we show this to
the art historians and say
do you think they match?
We don't try to tell
them this is the answer
don't ask me how I got it, which is
the typical scientific approach
to dealing with unscientific people right?
What I'm gonna show you
now, remember that map
I showed you earlier of
all the thread counts?
These are the ones that
match, there's the four.
Now what's really interesting is
they're all off by themselves.
So one of the things that they noticed
in Rembrandt's work when they were trying
to find canvases that matched was the ones
that had thread counts entirely different
from all the rest of the canvases
that the artist had used often were coming
from one roll, that's why
they matched each other.
It turns out the example I gave you at
the beginning of Van Gogh
is another example of that.
He only used a roll of jute
once because he hated it.
His brush didn't move across it
like it did across the canvas he was
most used to and he never used it again.
So if you just take all the ones that have
that low thread count they must be from
that one roll if they're
Van Gogh paintings.
Here's the others and their pairs.
These are the other three pairs
that I'm gonna quickly show you.
How do we know these
were the ones that match?
We didn't.
So what we have to do is do what I said
at the beginning, we have to find
all the ones that are
close, like this bunch,
and then compare each pair
and move them relative
to each other and see if
they'll end up matching.
Make sense?
It's pretty straightforward.
That's what's so amazing about it.
This is not complicated.
I hope you didn't come thinking
that I was gonna tell you something
about relativity or something like that.
I mean this is pretty basic.
Why?
Because nobody's tried this before.
It means the easy ones
are still left to do.
Perfect research problem, right?
So instead of this
being low hanging fruit,
do you know that expression?
It's easy to pick?
This stuff's already lying on the ground,
all you gotta do is pick it up.
Because the area is so new.
So let's look at these three other pairs
and see if it tells us anything.
This one tell us, or at least makes us
question something about the dating.
This is actually the first match we found.
And you're gonna say, well sorry,
I'm not supposed to do it that way.
Here's two paintings, we drew the
weave maps, do you think they match?
Well these look a little ugly
compared to the other ones.
Part of the reason is these
are very small paintings.
So the area in which we're doing
the little counts, they're bigger.
And so it looks a little cruder.
And plus this is in what's
called the weft direction.
For those of you that aren't an expert
in weaving and so on and
so forth, this is how
you keep non-experts out of your field.
You give names that nobody
else understands to everything.
So warp and weft are the two directions
in the weave, and warp is the long one
for the roll and the weft is the
short one that goes back and forth.
That usually is done with cruder threads.
So the thread count is not
that fine, a regular fabric.
But the experts, when
they saw this, you can see
that there's lots of little details
where the things are matching up.
I spoke to the owner of this just recently
when the Leiden Collection did its
rollout of their catalog, and he told me
he was ecstatic when he saw this result.
Well I guess so.
This gives you more evidence
that it's actually a Vermeer.
This is a Vermeer because the
yellow shawl is overpainted.
Most experts would prefer for this
to be considered not a Vermeer because
it just doesn't look up to snuff.
So this was the one that I
read about in the journal.
This is The Lacemaker in the Louvre.
Which is undoubted Vermeer.
So this was, if you will,
where this weave match,
with other information, can help you
do something about attribution.
How about this one?
Do you think that matches?
I mean I should've shown you a bunch
that didn't so you'd see how different
they are, but I think you can remember
that each of these that you've seen
seems to be different from all the others.
So you can see again,
here's these various,
what I look for are the stripes
that are, if you will, at the extreme.
The darkest red and the lightest colors.
They all line up.
These two are in the
National Gallery of London.
Some experts consider them to be
what's called a pendant
pair, in other words
the artist painted them to potentially
be hung as a pair that
together had meaning,
which meant often, and this seems to be
a thing that Rembrandt
followed, is when he was
doing a pendant pair, most of his are
pairs of a couple, the husband and wife.
And the canvas was bought
for both paintings at once.
So you buy one piece of canvas and cut it
in two and do the two paintings.
So at the same time it seems that maybe
Vermeer was doing this, but there's
some debate because after all, maybe he
bought one piece of canvas, but he never
meant for them to be pendant pairs.
But Walter Liedtke, who
I'm always working with,
this is one of the pendant
pairs in his catalog
and it has to do with
the idea that the theme
of these two has to do something with
one of them's the good
girl and one of them's
a bad girl, it's a moral
lesson which is basically
a lot of things you
find in Dutch paintings.
How do we know?
Well if you look at the painting on
the wall back here, that's a cupid.
And if you look at the painting here
which you can't see, it's a brothel scene.
So which one's the good girl?
Right?
So there are other hints as well.
Here's another one.
Unfortunately I didn't
find this one in time.
Walter died unexpectedly
in a train accident
a couple years ago before
I could show him this one,
but this one would be another one
you might expect would be a pendant pair
because if you looked at the, I know
this one's upside down, that's because
that's the way they matched.
But the model seems to
be the same and the theme
is similar, it's an
astronomer and a geographer.
And I showed this to Arthur Wheelock
who's another top Vermeer expert
and his reaction was yeah, well you might
expect them to be on the same canvas.
But the question of whether or not it's
a pendant pair is still
up to the art historians.
Walter said yes and Arthur said no.
The debate continues.
This puts your thumb on
the scale, it gives you
another piece of
information that tilts you
towards imagining that
they're a pendant pair.
Okay, so that's weave matches.
That's thread counting.
And now what I wanna do
is tell you something
about why this has to be
done in collaboration.
This is a cross-disciplinary
activity, it has to be
done in collaboration
with an art historian.
So it turns out, for
those three of you in the
audience that know about
Fourier transforms,
'cause I know two of
them sitting back there,
that when you tilt an image the Fourier
transform tilts the same amount.
So that means that when we were looking
at the threads, we could not only tell
how many there were per centimeter,
but also what their angle was.
Now we thought oh that's great!
We almost threw it in the trash.
We weren't asked to do that.
So let's give them a
picture of that, why not?
So why?
Because the reaction was oh
god look, there's cusping.
What's cusping?
I had no idea.
Well when you stretch a
canvas on the strainer,
and you pin it in in a
few places, it gets this
kinda scalloped, where
the fabric stretches.
So if you look really
closely here on the side
you can see this scalloping a little bit.
Over here, not so much.
And down here doesn't look like it at all.
That will tell you something else
about whether or not the
painting's been trimmed
or cut apart since the ground layer
and such was put on it, but let's not
worry about that for a moment.
The point is see that cusping there?
Because it gives you a
great deal of information,
we thought drawing the
angles would be useful too.
So what you get is stuff
like this where the angle,
so the light color's like
an angling pointing down,
and the dark color's like
an angle pointing up.
So what's happening right at this point
is where it goes from this to this.
So that's where the nail was.
So immediately from this plot, you can
tell how deep the cusping is because
it changes color, and you can also tell
where the nails or the tacks were spaced.
Turns out the art historians
wanted to know that.
And there's a variety of
reasons, and one of them,
you'd like to know something
about whether or not
maybe the painting was trimmed
at some point in its history.
And again, like I say, we
would have never done this
or had been interested in
it if the art historians
hadn't seen this picture and gone
oh we know what to do with that.
Okay, so that was one extra
thing we didn't expect to find.
So at this point we're going wow,
look, it tells us about this, it tells us
about that, this is pretty good.
So let's do all the Vermeers.
Which is, I showed you, where we ended up.
But there's one more surprise left.
So when we were doing the weave maps,
I don't know if you can see this
right here where the big arrows are.
This, by the way, again is 14, 18,
19, and 31, it's the group of four again.
Or I tend to call it
the gang of four, okay?
Chinese reference I guess right?
So see this right here?
I saw this in the map and I call up
my colleague Don Johnson at Rice
and I said Don, you screwed up.
The program's not working right,
there's this mess in the middle,
I don't know how you got that.
Can you fix that?
So we're back and forth on the phone
for a few weeks until we finally
decided to look at the x-ray.
So what would this indicate?
Remember these little
wiggles and color mean
that the fabric is doing
something like this right?
It's changing its angle, okay?
This is in the middle,
this is not near the edge,
it can't be cusping like these things.
So you see cusping in these, this is
the horizontal thread weave map.
And this is the vertical thread weave map.
Over here the things are doing like this,
and here they're doing like this.
So what causes that?
Well this is where you've
gotta know something
about weaving, but essentially in the
weaving you have the long dimension,
this is the warp, and you have
the short dimension that's the weft,
you open and then you push the shuttle
through and then you close them
differently and you push them back.
That's what gives you the one over
and under type of pattern.
But the problem is if
you don't give yourself
enough length on the tread when you
pull it across, then what'll happen is
it's not long enough for
the thread to do this.
See what I saying?
In other words the thread
that's coming across
has to go over and
under every one of them.
So if I opened them up and just put it
straight across which would be my
engineering tendency, pull it night
and taut, don't waste any thread.
As soon as I actually
close it down and push it
together, what happens is it's too short.
So it pulls the threads
a little bit together.
Which means the ones you've
already woven go slack.
So that's where you get this thing that's
showing you the slackness in the threads.
Engineers wanna give
everything a name so you know
what it is, so I thought
this will be cute.
This is, it looks like a snake
going through the grass right?
It makes this pattern.
And what we discovered is that
can only happen in the weft direction.
Now you're supposed to give
me a really big so what,
but we'll be back to that
in a second all right?
So the reason is because the curators,
and especially the
conservators, would like
to know for various
reasons which direction
was the warp and which
direction was the weft.
And it turns out that it's
almost impossible to tell.
The really only forensic that works,
and again, I think I'm
gonna get more hands up
this time, how many
people have done sewing?
- [Audience Member] Sewing?
- Sewing.
You go to the fabric store,
you buy a piece of cloth.
What's on the edge of the piece of cloth?
Doesn't look like the rest of it,
it's this kinda ugly looking stuff.
- [Audience Member] Salvage.
- Salvage!
Gee, somebody knows the secret word.
So what happens is that
over time the salvage
is on the corner of the painting, but it's
the edge of the painting as it goes
around the strainer
that gets the most wear.
It starts to deteriorate, so they cut
the painting off, the
face of the painting off,
and then glue it to a
bigger piece of fabric
that then goes around the strainer.
That's why the backside of the painting
you don't see the canvas anymore, it's the
lining canvas that's been adhered to it.
That's the reason, by the way, for x-rays.
I didn't tell you that at the beginning.
Why, somebody should've
asked, how come you don't
just turn the painting
around and take a picture?
Because the threads are on
the back, just count them.
Well it turns out they're not.
The threads you see on the
back at the lining canvas.
But when you make a painting and
your canvas is rough, so the first thing
you wanna do is make the surface smooth.
What were they painting on before canvas?
Wood.
So you made a panel, you
made it really smooth,
some special artisan made that panel
and made it very nice and smooth
'cause it's easy to
paint on if it's smooth.
In fact it's difficult to paint something
with fine detail if it's not smooth.
So the first thing you do to prepare
the canvas is lob on some cheap paint
that you got lying around,
typically lead white
because it's typically the most
plentiful, cheapest thing you've got,
and it goes in-between the threads.
Some of it sits on the top,
but the top of it is smooth.
The point being is that
the top of the threads
only has a little bit
of it, and in-between
there's a lot of more
that's sitting in there.
So what?
Well I said lead white, it's called.
What does that mean when
you hit it with an x-ray?
It's opaque to the x-ray right?
The lead absorbs the x-ray.
So the parts that are
thin, the x-ray's gonna
go zooming right through
and it'll expose the film.
The parts that are really thick,
not much of the x-ray gets through,
and that's what gives you those
pictures I showed you earlier.
Like this.
That's why you see this.
What you see is not the threads.
What you see is the impression
of the threads in the ground layer.
Okay?
If we look at these angles here
from the x-ray and everything, we assume
that there's some wiggle going on here,
well this is what it
looks like in the x-ray.
Can you see it?
Well I've turned the painting up so that
the actual weft snake
is in here somewhere.
And if you look really
closely, there it is.
You can see some wiggling.
Now imagine trying to
find that in a big canvas.
It's not easy.
But if I give you that picture
it tells you where to look.
So we call these things
weft snake indicators.
Isn't that cool?
So it turns out this was something,
this now becomes the
only confident method,
short of salvage, to tell which way
is warp and which way is weft.
But the problem is it's
not in every painting.
Only about 40% of the paintings
have these weft snakes in them.
'Cause it's a weaving flaw.
You don't want it, it shouldn't happen.
But the reason you know it's a weft snake,
in fact, is it goes all the way
across both of these paintings.
In other words it should
be something that traverses
the entire weft direction of the canvas.
So we know that this now
is the weft direction,
this is the warp direction, and this is
probably the dimension of the canvas roll.
So we have a bunch of extra
information that comes from that.
But again, this is an indicator,
there could be other things like
other types of flaws that might
give you that behavior, so you always
have to look very closely at the x-ray.
Again, looking closely is what
the art historians get to do.
All we're trying to do is give them
some tips about where to go look for them.
So we started out counting threads.
And we ended up finding three things
that we didn't expect to find.
One of them was these maps with stripes
that allowed you to
decide if two paintings
might have come from the same canvas.
And we just saw that we had a really
terrific way to look at cusping.
If you remember, sorry, I hate it when
people flick the slides, but I'm the one
in control of the clicker, so here we go.
Oh by the way, the
cusping is so slight here.
But it's unmistakable when
you look at this image.
So basically it enhances your
ability to see the cusping.
And in the terms of the weft snake,
it gives you an indicator of a flaw
that would be very very
hard to locate in the
canvas itself, it would
require an enormous
amount of time hunting for these things.
Okay.
So now we come to the part that Stephen
told you I would talk a little bit about,
the painting here at the Frick.
I know it as L21, but you guys
know it as Mistress and Maid, okay?
So what we're gonna do, where's Ellen?
Were you passing out
pieces of paper Ellen?
Everybody has, hold it up!
Everybody's got, it's in front of you,
so what we're gonna do
is we're gonna send you
home and you're gonna become a scientist.
So what you're gonna do is you're gonna,
here's the piece of paper
that was handed out.
So what you can do is play paper dolls
and try and find out if these match.
I'm gonna give you the instructions.
They're on the next slide,
but let's do it verbally.
This is the horizontal of
the one we're interested in.
So this is the painting
that is owned by the Frick.
And here are the ones that showed up
on that plot of the ones that were similar
in terms of average thread count.
So what we're gonna do is we're gonna take
the horizontal of this one and we're
gonna use the color bar that goes
with that, what do I mean by color bar?
The colors are associated
with thread count numbers.
So we'll use that same
color bar on the horizontals
of the three that may be similar to it.
And then rotate the vertical ones
and use the same color bar.
So clearly this one's
off, this one can't match.
I mean some of these are obviously no go.
And you can expect that
because if you look
at the thread counts, they're quite
far apart, there's 17 and 14.
And here this is reversed,
about 14 and 16 something.
So you wouldn't expect these two,
or this one, to match that.
But now the question I'm gonna ask you
is do any of these match that one?
This one doesn't look, I
mean it looks a little dark.
So at this point it's a matter
of playing with it, I'm not gonna
tell you the answer 'cause I don't know.
I mean it's a question of actually
arranging them in the right way
and then all of a sudden it's like
putting a puzzle together, it fits.
So you get to discover
whether or not there are
any matches there, please tell me first.
If you find one.
This is, if you will,
and it has to match in,
it can't match in both
direction so to speak
because it can only be
aligned this way or this way.
All you're gonna find is
that one of these four,
one of these two matches this one,
or one of these two matches that one,
or there's no match whatsoever.
So on this row, only one of these
can match the one it's associated with.
No more than one.
So the question is does
this, do any of these two
match this, or these two match that?
Do either of these two match
this or these two match that?
Or either of these two match this one,
or either of these two match that one?
Anybody know the answer yet?
I don't know the answer, I really don't.
I mean I think they don't match.
Again, we just finished
collecting all these things.
Peggy told me it was okay to say
something about the book we're writing.
We're just about to finish a book on
Vermeer's canvases that has all these
weave maps in it and
all these comparisons,
but we haven't had the
chance yet to go through
all the potential matches
and see which ones
we can discover beyond
the six we found already.
So in case you prefer to read instead
of listen to the instructions, this is
what you're supposed to
do to become an expert
in art history, and the instructions
should be on the back
of the piece of paper.
Read them before you cut it apart, okay?
Because then it'll be hard to figure out
what the instructions
are, does that make sense?
(laughter)
Is that fair?
You didn't expect to be asked to
crowd source science here did you?
But this is what's going to happen.
As computation becomes more and more
of a standard tool in art history,
there will be more and more of these
types of things that will actually
extend the reach of the art historian.
And in my opinion,
deepen our understanding
of the paintings and actually make them
even more valuable and resonate to us
than they are just from the viewing.
And at that point I'm
going to say thank you.
(applause)
I don't know what time it is.
- (mumbles).
- Oh, that's a little long.
- Thank you so much.
- Yeah.
- All right, thank you very much Rick.
That was really great, and I think now
we're all experts on weave counting.
And we're all looking forward to
your book coming out later this year.
I know it'll be helpful for my research.
I wanted to begin a little
bit with your background.
Why are you, someone's who's trained
in electrical and computer engineering
and applied mathematics,
why have you chosen
to apply your knowledge to
the field of art history?
- 'Cause it sounded like fun
would be the easy answer.
Actually, a lot of engineers
read mathematics for fun.
I know that sounds grueling, but I tended
to read art history
journal papers for fun.
So for many years I thought wouldn't
this be a cool thing to do.
Those of us that do technical research,
things move so quickly that during
your career actually, the topic you're
working on collapses or everything's
solved and you need to
move into a different area.
So that had happened to me once previously
in my career, and I'd
reach that point again.
It tended to happen about every 15 years.
So finally, 10 years or
so ago, I decided okay,
I wanna pick a problem
that I'm emotionally
involved with rather than something
the federal government will pay for.
I mean that's a little crude,
but that's kinda the truth.
So I thought okay, I'll see if there's
anything I can do, and
even if there isn't,
I'll have had fun because I get to go
to museums and look at paintings.
How bad can that be?
So I'm gonna plug Cornell
a little bit here,
so I wrote a note to
my department chairman
and dean saying hey guys, guess what?
I'm gonna be a historian instead
of an engineer basically which means
no research money, no publications,
and no grad students for a while.
I know some of my colleagues
that have tried this
at other places and they
get all kinds of grief.
You can't do that, you can't do that,
especially if you're
untenured, we'll fire you.
This kind of stuff.
Both my department chairmen
and dean said to me that's great.
If this is something that you can do
that's unique and people know you're
C-O-R-N-E-L-L, if people know you're
from Cornell, then that's
a good thing for Cornell.
So I've been very encouraged.
Cornell has a very strong history
of interdisciplinary research.
So I ended up happening
to be in the right place.
And I got really lucky very quickly.
I got in the Van Gogh Museum.
I mean that's the great
thing about the Dutch,
one meeting and they decided
to commit to this project.
They told me to leave the room, and then
the three of them voted,
and I came back in
and they said you're in, we're gonna do
whatever it is and so and so right?
It's been a series of accidents like that
where I just ended up at the right place
at the right time because
if I had tried this
20 or 30 years ago it
would've been a big flop.
There was no way to digitize images,
the computational ability wasn't
quite as easy as it is now, and I think
also you'll notice digital humanities
is showing up in the
newspaper all the time.
So I'm hitting it right
before it takes off
and that's a marvelous time to be
involved with a new topic.
It was born out of my
love for hanging around
with paintings, but it actually
turned into something much bigger.
Is that a fair answer?
- Absolutely.
And I just wanted to probe
that a little further.
Although your analysis, they focuse
not exclusively on Dutch artists,
but they do, you do
tend to focus very much
on Dutch artists, is
there a reason for that?
- It's mostly, that's where I wanted
to start 'cause that was my interest.
But because, and this is where I guess
I'm plugging the Netherlands, they seem
to be a few steps ahead of everybody else
in moving into this area
of digital art history.
So they have national institutions
that share information among the museums
and they were very eager
to get into this area.
So by accident, since the Van Gogh Museum
was the first place I went,
or the first place I had good contacts.
Though it's also been, I have to say,
let me plug some New York City museums
that have been fabulous to me.
One of them's MoMA.
We did photographic
paper texture similarity,
so there's a lot of different
things, that's one of them.
The Met has been marvelous and working
with Walter was just fabulous.
And Morgan Library because
now my major interest
is with paper, and so
I'm working with them
because that's a big
part of their collection.
I'm not sure that's a direct
answer to your question.
- No, absolutely it does indeed.
But I wanna also ask you,
if you can share with us
some more of your significant findings.
I know for example Velazquez
has figured into your work.
- That's a Frick painting.
- That's right, I thought you might
tell us a little bit about that painting.
- Should've mentioned that one.
If you haven't seen it, go see it.
It's a three quarter portrait of the king.
- It's in our west gallery.
- Sorry?
- It's in our west gallery.
- West gallery, that I didn't know, sorry.
It's a fabulous painting.
It turns out that it was
painted not in Madrid.
It was painted in Fraga, which where
there was a big battle going on,
and the Spanish were going to win.
So the king goes to visit the site,
and to commemorate that, and also
to have I think a big portrait for
the victory parade back in Madrid,
I'm not sure I've got all the facts right,
but anyway, the king is painted
and a courtier is painted by Velazquez.
Now he painted, he liked to paint
a courtier before he painted
one of the royal families.
Essentially they had, it turns out
at that time that the courtiers
in the 17th century court in Spain,
many of them were dwarfs
as like a jester factor.
Oddly enough, the documentation for which
dwarf it was had been burned in some
fire in the archives or
something like there.
So there had been this debate going on
for 20 years about which dwarf it was.
And I happened to be talking, where's Joe?
Maybe he had to go, Joe Godler,
I had had some contact with the Frick
at that point, and I was talking
to the Frick staff, and it turns out
you had a visitor from Spain.
I told him about the
weave matches and stuff,
he says oh god, he says I
know which dwarf it was.
He says I did all the documentation
research, it's this one.
And I said well get us the x-rays.
So it turns out they matched.
So that was my second
paper in the Burlington
Magazine which is, now
I'm plugging myself.
So those of you that don't know,
the Burlington Magazine considers itself
the top art history journal in the world.
So I've had two papers there, one was
the Van Gogh stuff, and one of them
was this paper about Velazquez.
Because we did Van Gogh, we started
doing all of the Impressionists.
So we found weave maps with Matisse
and Monet and so on and so forth,
but because the projects got so big,
that part was peeled off and given
to one of my colleagues and he's now
doing weave maps for the Impressionist
collection at the Art
Institute in Chicago.
If you go to their online catalog
you can see all these weave maps.
So there's lots of matches
it turns out, but in
each case you have to
reinterpret what it means
if they're there, but what it means
to each artist is a little bit different.
- [Margaret] Great.
Okay, I wanna go back to Vermeer.
- Okay.
- So we still have a lot to learn
about Vermeer's relationships
with other artists,
working in Delft during his lifetime.
So how could thread count analysis be used
to expand our knowledge
or possible collaborations
or other exchanges among these artists?
- So I'll start with
the negative part first.
Let's imagine that you think two people
shared a canvas and so
you match the weave maps
of their paintings and they don't match.
That doesn't mean anything.
There's no reason for them to match unless
they were cut from the
same piece of canvas.
So not having a weave match is like
having a count match, because it doesn't
really mean much which means it's
possible that they were similar.
Not having a weave match doesn't mean they
weren't done by the
same person or anything.
But we have noticed, we
haven't done enough yet,
in fact when the project started
it was called Vermeer and the Delft School
when we started because we were gonna do
as many as we could of
contemporary artists
from that period, and
the thing we ran into
right away is nobody does x-rays
of paintings by secondary artists.
Not at least x-rays that
cover the full painting.
So it's that lack of data.
But I can tell you one
interesting thing with Van Gogh
is that we found a Van Gogh that
matched the weave of a friend of his.
We thought wow, that's cool.
Then they went looking through the letters
and there's a letter to his brother
that says my friend and I went to the shop
and got canvas and went painting together.
So it becomes a way to
hook these together.
I know that there was a lot of sharing
among the Caravaggisti in the Netherlands.
So that's another project
that's been proposed.
But if you're willing
to think about big data,
another thing that's in the newspaper,
imagine being able to do
weave maps of all of the
paintings in the Netherlands
from the 17th century.
If you did find weave matches, it would
tell you unexpected things probably
about who was collaborating with whom
and who was sharing materials.
But it's not something that answers,
I've said this before, not something
that answers the question by itself,
but becomes a very useful piece of info.
I think of myself kinda like a detective.
It's the lipstick on the
glass on the mantelpiece
and not the footprint in the mud in
the garden outside the
window that matters.
But I don't know that in advance.
So what I've gotta do is collect
all this information, and then later
you'll take the pieces that fit together
in the puzzle that go with the other
documentation and the stylistic analysis,
and the materials analysis of the paint,
and so on and so forth.
Does that answer?
- [Margaret] It does indeed.
- Okay.
- I wanna actually now
discuss the potential
limitations of third account studies.
We'd had a conversation previously,
and during that you
mentioned that you would've
been very eager to use your thread count
program to study Rembrandt's paintings,
but that his pictures
offer specific challenges.
Can you comment?
- So one of my colleagues said
that Rembrandt used crappy canvas.
So what that means is the regularity
of the weave is not that great.
He may have used either low cost canvas,
it's not clear, but
the point is the method
we're using now mathematically falls apart
if the weave is not really quite regular.
You'll get ridiculous answers.
So that means that, now
that we know this matters,
that's the challenge to the engineers.
Give me a method that works on
the canvases that are somewhat irregular.
In other words can you
come up with a technique
that automatically marks each thread.
Because if you could do that then
it wouldn't matter, you wouldn't have to
use the Fourier transform
business for instance.
That actually was part of my
strategy from the beginning.
I see myself as trying to find these
problems that art historians care about,
that have simple engineering solutions.
So what I try to do is use the simplest,
dumbest solution I can think of,
knowing that it's not the best one,
but it's good enough to show results
you didn't expect, which
is what we just saw.
Then I turn around to
my engineering friends
and I say hey look,
these people care about
this problem and I used a Fourier,
you used a Fourier transform?
That's a dumb thing to use.
I can do better than that.
And they're hooked.
So they want to now show that they
can come up with a better one.
And that's what's going to happen,
there are already people trying to do,
want to do thread level marking
and a whole bunch of other stuff
to take it to the next stage.
So right now, and this is something
you have to know if you're trying
to use these things, so those of you
in the audience that are curators
and conservators and such, you have to
understand when somebody's
using computation
what the limits are of their tool.
I know this might not be
easy, but make them tell you.
They're not gonna want to.
They're gonna try to tell you
it works all the time perfect.
It's not true ever.
So part of the things
that needs to come along
with this is what you're asking me about.
Is can this work for
everything, and the answer's no.
I didn't mention, my personal story was
I didn't go to my first fine art museum
until I was in my early twenties.
I didn't grow up in a part of the country
where that was a normal thing to do.
The first museum I went
into, I went in the room,
it was the Dali in Berlin, I went
into the room with the Rembrandts
and I'm there for a couple of hours going.
So that's why I wanted to do Rembrandt.
So we tried and the Fourier
transform method didn't work.
It was working great with Van Gogh
and the Impressionists and even
Vermeer, that's why we did Vermeer.
So that's why now my current project
is trying to identify watermarks
in Rembrandt's etchings 'cause I finally
wanted to get around to Rembrandt.
- Actually that's one of my
questions for you actually too.
So you are now using
your computer programming
wizardry to study watermarks on drawings.
How, can you tell us a
little bit more about
how you're designing
your program to do this?
- It's a completely different technique.
It's a different method.
How many people know, I mean everybody
knows what a watermark is right?
Yeah, you hold up any nice piece of paper
and you can see it's a little thinner
there so you can see the shape.
It usually indicates the manufacturer
or something like that.
So let's imagine you're
in the 17th century
and you make a piece
of paper that everybody
loves and you have a certain watermark
in it, what do you think's gonna happen?
The competitor's gonna
make it cheaper with
the same watermark and
sell it like it's yours.
Sound familiar?
Okay, so that means there's a lot
of watermarks that are very very similar.
But watermarks are
handmade, they're pieces
of wire that are bent and then sew into
the screen that you use
to make the paper on,
if you've ever handmade paper.
It's like you dip the screen into this vat
of spitball I guess, I mean that's the way
I think of it, the
slurry of water and pulp.
And when you lift it
out of there the water
drains out and leaves
the pulp on the screen.
Where the pulp is sitting on the screen,
it's a little thinner than the rest
of the paper and so that
leaves an indentation,
that's why you see the watermark.
And the watermarks can be very similar,
so how do you tell the difference?
Well it turns out
Rembrandt's so well studied,
and I have an appointment
at the Rijksmuseum,
and the number one
person in the world that
knows this stuff happens
to be on the staff there.
So I was asking him, hey Eric,
how do you tell which one's which?
So he spent six years
identifying them all,
500 and something different
variants, without a computer.
He just did them all by hand.
Think about it.
So what he has in his
head is he knows if he
looks at this one he
should look at this place,
and then if this is similar or shaped
like that it's this kind
instead of that kind.
So what he's got in his head is
something called the decision tree.
Everybody here's played
20 questions right?
Is it bigger than a bread box?
Is it alive?
This kind of stuff.
So that's what we're trying to do is we're
trying to figure out
what the decision tree
would be so that we could end up with
every one of the different variants,
and you would only have
to ask six or seven
questions and it would
say well it's this one.
And if you ask any expert that tries
to do this now, it's a slog because
you've got a three volume book that you
have to look through, even for Rembrandt,
and it take hours to be confident
that you've gotten the right one
and we can now do it in five minutes.
So that means that you'll be able to take,
once this is finished, this is by the way
such a fabulous student
project, we're going
slowly to allow students to participate
and build the tree, not us.
So once we're finished it'll be free,
it's gonna probably be archived at
the RKD Dutch Institute for Art History
documentation in the Netherlands.
And everybody will be able to upload
the image of their paper, their watermark,
and then the computer will come up
and ask you is it a horse or a tree?
So you pick that, you press the button,
and then it says does the horse have
a so and so, or is the rider, and so you,
I don't know what he's talking about,
so we're gonna mark the
part in the watermark
that you're supposed to be looking at
which I think's the most important part.
So we mark the part and then
you'd answer yes/no questions.
And after half a dozen yes/no
questions, you have the answer.
And two of my colleagues here from
Cornell Tech are helping me with this
because it turns out in the 1970s
this was a big deal mathematically.
But we have an extra problem is
we're finding new watermarks.
Because Eric only looked at the
prints in the major collections,
so we're trying to look at prints
in smaller, mostly university collections,
and a colleague from
Cornell who's working,
Andy raise your hand, (mumbles)
who's a curator at the Johnson Museum.
And Vikram and Sujay, you wanna
raise your hands?
They're at Cornell Tech, they're the
engineers that are working with me.
So we eventually hope to be able,
through this student project, to build
a piece of software that we'll share
with the world that'll allow everybody
to identify their Rembrandt watermarks
in the entire collection,
private collectors
as well in a few minutes.
Watermarks can tell you a lot because
the assumption is he only used
a batch of paper with this watermark
at one particular time
and never used a batch
of paper with that same watermark again,
there's so many different watermarks.
So the watermark becomes a very
terrific way to date the paper.
Unfortunately only about a third
of the prints have watermarks in them.
So we're also working on trying to look
at the screen pattern, because the screens
are handmade, and look at that and see
if that gives us information about
how to say two piece of
paper were moldmates.
They were made on the same mold.
Just like two pieces of
canvas are rollmates.
They're on the same roll of
canvas, isn't that clever?
So yes, that's my latest, biggest project
and one that I'm deeply
emotionally involved with
because after all, it's Rembrandt.
- Excellent.
Before we open up to the audience, I just
wanted to go back to the limitations
of third account analysis again
and just ask isn't it possible,
for example, a member
of an artist workshop,
to use a canvas support that's in
the workshop for their own work?
- Of course.
- Alrighty.
Has the possibility of
workshop interference
influenced your choice of artists?
Vermeer and Van Gogh for
example have no recorded pupils.
- Actually it's an accident.
- Oh okay, a lucky one.
- A lucky one.
Remember, I tried Rembrandt.
Rembrandt had lots of students.
This is why I keep saying this is
one forensic out of many, so then
you need all this other information
about who they worked
with, if they had students,
did they share studio space,
did they share materials.
And the odd thing about Van Gogh
is we're just loaded with information.
Go online, read his
letters, they're fabulous.
They just tell you so
much about what he was
trying to do and how he was doing it.
But they also have buried in them
every note he wrote to his brother,
I'm out of canvas, please,
this kind of stuff.
So we know when he got
his canvas we think.
So that gives us a clue.
It's that information that you
have to put together with it.
So sure, it doesn't bother me if maybe
it was somebody else in the studio.
That's why when I show the two
I don't say this proves this.
All I say is hey look, these two look like
they're similar, what do you think?
Now that I tell you that they're similar,
what are you gonna do with that?
Well what you're gonna do is you're
gonna bring in lots of other pieces
of information to actually make
an art historical judgment
about authenticity.
Or artist intent, whether or not they were
intended as a pendant pair, et cetera.
So it wasn't by design.
It was another lucky accident,
but on the other hand you wouldn't
be afraid of doing this for somebody
that shared materials as long as
you had other information about that.
This will be an important forensic period.
It's just that by itself it needs
to have all these other things, okay?
I mean after all, it's not just
the lipstick on the glass, but it's
also the cigarette butt in the ashtray.
This kind of thing.
- Thank you very much, Rick.
I'm gonna now open it up to the audience.
I'm sure you might have questions as well.
- And there should be
microphones floating around.
- They're gonna be coming
around to give you a microphone.
- First of all, thank you very much.
My question is the process seems to be
to take an analog piece of information,
an x-ray, to digitize it, and then to
make it analog again by visualizing it.
Is there no way to use
the digital footprint
of the resulting information that you have
to then analyze that in a way that
it can find what matches what and in
which way rather than going back
to this analog-enclosed observation
process of trying to find picture
puzzle pieces and put them together?
- Yes.
You want me to say more?
There is a way.
- [Audience Member] Say more.
- Okay, so in fact that's what
we did to find the first three matches.
So it turns out again that it can
be done with a math tool that we
teach every junior in engineering.
So basically what you
do is since the stripes,
let's take vertical stripes, and since
they should be basically the same color,
you average all the counts and columns,
which gives you some sort of profile,
and then you match the profiles best.
Think of sliding the two
curves next to each other.
All of that can be done mathematically,
it's called correlation, so it's another
simple mathematical operation and that's
how we found the first three matches.
The problem with that is that these
are very very noisy images in that regard.
So it's a rather
imprecise way of doing it,
there might be more
precise ways of doing it,
but it's not something I
can tell the art historians.
It's not something I can
say I did all the math,
trust me, this is the answer,
don't ask me how I got it.
So what we're reaching for in every one
of our projects is to somehow turn
what we found into
something that can be turned
into something visual
that the art historians
can recognize in what they're looking at.
The last thing I wanna do is come between
the art historian and the object because
I wanna make sure that their knowledge
gets used in the final analysis.
So yes, almost all the
things I'm talking about
could be done totally start to finish.
I thought you might even ask why are we
using analog x-ray films right now.
Digital x-ray machines that are available
for museums don't get
the resolution we need.
We need about 600 dpi.
So that's why the analog
films are actually
much more precise than a digital image.
The other thing that I did when I started
all these things, I'm only gonna work on
a problem where the images
are already collected.
The last thing that's gonna happen
is I'm gonna go into a museum and say
man I've got good stuff to tell you,
but the first thing you're gonna do
is take all the paintings off the wall
and ship them to my lab so I can,
it's not gonna happen.
So I had to find places where they
already had the images, and every museum
has drawers and drawers full of x-rays.
Because they're used for something else.
They're used to see the painting
that's been painted over where the arm
was like this and now it's like this
and that makes a big difference.
They have these things and that's what
we were using, but yes, what we'd like
to do, especially if we go to the
big data end of things, is we'll have
to come up with computational methods
that get us so close to the final answer
that you only gotta choose between
two or three things to
figure out what's going on.
But here I thought it would be nice
to show you what it means visually
and that you can
actually, from these maps,
affirm to yourself that it looks
like a match or doesn't, so that was
the reason for handing
out the pieces of paper.
Is that a fair enough answer?
Okay.
- Are there explanations in
the gang of four for the...
Disparity in the dating, or an explanation
for the new dating that you know about?
Particularly I guess the
Woman With A Lute in the Met.
- Yeah.
- Which is the one what would be
particularly earlier in the dating.
- Now the woman With A Lute in the Met
and the other two, which I can't
remember the names of, L18 and L19,
are all about the same date.
It's the one in the Dublin
Museum that's six years later.
I have no idea how that's gonna shake out.
We had a paper that was in the
Metropolitan Museum journal, and Walter
made comments that I could read to you,
but essentially you question what's what,
but that's gonna require an enormous
amount of other information to resolve
that question, what it means.
Do we need to re-date these things?
Who knows, maybe the guy
was actually saving canvas.
We know that when he died there was
an inventory taken of
the things in his studio
and there were 10 canvases
primed and ready to be painted.
He only painted 30-something.
Why has he got 10 left in his studio?
We don't know anything about how
he bought canvas, but there will need
to be more information
brought to the problem.
It's just that this
starts making you think
maybe we should reconsider this.
So but stylistically I think almost
everybody dates them as
being quite separate.
So not a very satisfactory
answer, I'm sorry.
I don't really know, it's beyond
my expertise to make those guesses.
Only two questions and that's it?
I wasn't that clear.
(laughter)
Pepe.
- Hey there.
Thank you, that was great.
- I'm glad you said so.
- The two methods you've described to us
both look at the
structure of the supports,
I mean maybe this just goes beyond
the borders of what you're doing,
but could one look at the composition?
Could you do
chromo-spectrographic analysis?
- Yeah, there's all kinds of
different things you could do.
I mean the one thing you can't do from
x-rays is decide what the materials is,
is it linen or hemp or whatever.
There would be many other
different types of analysis.
The most important one
seems to be at this stage
to determine what the
ground layer materials are.
Because often that's something that
the artist does over
and over again the same.
Not always, but in certain
periods it tends to be,
I mean you know this better than I do.
So we were limited to the x-rays.
And we were asked to do thread counting.
Yes, there are many other possibilities.
For instance, one of the ones that's
being considered now for our paper project
is trying to use stereoscopic imaging
to try and see indentations to help
identify the watermarks without
having to use a beta-radiograph.
Yeah, I mean I can only go so far
with the little bit that I know.
I just hope I get everybody excited
enough to do the next step.
That's basically my goal.
- [Pepe] Great, thanks.
- [Audience Member] I have a question.
- Okay.
- I'm wondering if you're willing to share
your technology with
other museum conservators
around the world if the idea of doing this
could spread out to German
artists, French artists,
et cetera, so technologists
in those countries?
- Again the answer is yes.
(laughter)
So the book that we're going to publish
will actually have free software in it
that runs with a free downloadable reader
from Mathematica which is a company
that makes a computation package.
Our interest all along was to try
and give this away to everybody.
We'll be in the position,
once this software
is available, you'll be able to download
and use it, but my colleague Bill Sethares
at University of Wisconsin has done all
the programming, and
basically either one of us
is probably willing to show up at
any museum and teach you how to use it.
Because it won't happen otherwise.
So yes, in fact we've done things,
I've been doing things like this
over the past several years.
I had CREST Foundation
money for a few years
to go around to the conservation programs
in the US and the Netherlands teaching how
image processing could be
used for various things.
A few generations of currently graduating
conservatives have already been taught
how to do all this thread counting.
The software however in the past has
required you own some really complicated,
expensive numerical computation package.
Recently Mathematica
introduced this reader
that you can download for free, so we'll
give you the software to go with it
and it turns out you can actually
use it on your own paintings.
In another five to 10 years I would expect
these things would sit
in the web for free.
So there will be software, I know that
the Rijksmuseum, Rob
Erdman of the Rijksmuseum
is trying to develop things like this.
So there are a number
of us that believe in
open software in the sense that you
don't charge people to use it.
Most of the people I've worked with,
in fact everyone I've recruited to work
on projects with me, they have to
promise that that's
what's going to happen.
That there's no sense of commercialization
and there's no sense of tight-holding it
to yourself and being the only expert.
So this will really only pay off
if it's done on a large scale.
The only way it'll be done
on a large scale is if it
becomes free and widely
available and trivial to use.
- Sorry, I don't wanna
monopolize, but just
to make you a little
bit crazy, as an artist
I am very aware that different
brushes have different signatures.
They of course age and
hairs fall out and whatnot,
and it certainly wouldn't, it may not be
applicable in an artist whose work
is very glossed and
very fine and detailed,
but nevertheless I
think we can all imagine
a very impasto, very thick, goopy artist
brushstroke and wondering has there been
any attempt to look at brushstrokes
as identifiers of a particular time
because there's a limited amount
of time that a brush will last?
- So first thing I've gotta do is
give you $5 for asking that question.
(laughs) So turns out that was
the first problem we looked at.
So the first paper I wrote was, we had
five or six teams that tried to
identify the difference been Van Goghs
and fakes by looking at the brushwork.
And you mentioned many of the reasons,
the impasto in Van Gogh's
paintings is very thick.
What I discovered was that was
the wrong problem to start with.
Now why am I saying that?
First, it's really difficult.
Partly because it's not like each
brushstroke is cleanly done by itself.
You're overlapping and you're hatching,
so it turns out from the image
processing viewpoint it's very difficult.
The worst problem is that I can't say
to an expert this is a fake and this isn't
because if they disagree with me
they'll quit listening to me.
So I knew that to get my foot in the door
I had to work on things where there was
an answer and everybody could agree to it.
Count the threads, it's
17, it's not 45, look.
So when I tell them the computer said 17
and they check it and
that's what they see,
there's no problem with absorbing what
I'm trying to tell them, but if I come in
and say I did all this
fancy brushwork analysis
and I can tell you that this is
a Van Gogh and this isn't, and they
think differently,
they're not gonna bother.
It's just not that
conclusive, especially if
I pick one that actually was a Gauguin
instead of a Van Gogh which is what
happened in some of the
presentations I've seen.
Because these things aren't perfect,
but actually the brushwork is something
that a number of people have looked at.
It will be very easy it you just Google.
You'll find a set of papers
since about 1970, sorry.
2007 there was a workshop,
that was the focus,
and a lot of teams have done
that here and in Europe.
It's relatively inconclusive still
because the problem is so difficult.
- [Audience Member] Yeah,
I have another question
actually about open source with material.
You learned how difficult it can be
to get digital images of x-rays.
Is there any possibility
of having the x-rays
that are available to you made available
for research on questions
beyond thread count?
- Okay, that's another one that
I recognized early on as important.
Or if you look at it the
other way, if I wanted
to protect my control of this topic,
I would never share an
x-ray with you right?
So it turns out if you
have a film x-ray you just
gotta put it in a scanner and you're done.
So now I've digitized it
without really high tech.
But yes, you can actually go to the RKD,
there are four Vermeers and six Van Goghs.
Full painting x-rays
available, so if you wanna
build a new algorithm, you
have a source to do that.
The Vermeer project that
we're working on, we're trying
to get all the museums to allow
us to post all the x-rays.
There are a number of museums
that are starting to do this.
The RKD in the Netherlands actually
sees itself as now a repository not just
for documentation, but
also scientific images.
And a lot of museums are starting
to put this information online so that
anybody can use it, and none of this
was happening when I started, it was
like pulling teeth to get access.
I mean I probably spent
85 to 90% of my time
the first four or five years going around
and playing nice with the museums.
Trying to convince them that I wasn't
some flake or that I wasn't going to say
that they did something stupid with their
paintings or whatever
they were worried about.
But the idea of sharing
this scientific data
is catching on, and I think in another
five years or so you'll be flooded
with these things if you wanna have things
that you can use without
having to go through
this begging exercise that I went through.
But that's important, it
won't happen without that.
So that's a good question.
Behind you.
And there was a hand over here as well.
Don't wanna leave people off just 'cause
they're sitting in the wrong place right?
- [Audience Member] Have you and your team
considered finding, actually
applying this method
for the textile collection attribution?
- Yes.
(laughter)
I mean these questions
are pretty easy right?
The problem there is that I've gotta find
the right people to work with,
and we need to have the right images.
Textiles present a much harder problem.
And the reason partly
is that imaging them is
difficult because they're
not necessarily flat.
You have to somehow control them, okay?
But the other thing
is, the sneaky thing is
that almost all paintings,
not all, but most
have this simple weave,
one over one under.
Not so about fabric.
So the problem is that
our analysis will require,
again, more complex tools, we can still
do it with a Fourier transform,
but interpreting it becomes much harder.
We've looked at twill, and we've
looked at herringbone a bit.
We know we can do twill.
There are some artists that have used
twill canvas, Caravaggio comes to mind.
But to go into fabric where you might
have an elaborate type of weaving,
we don't have a mathematical
technique to do that
right now, but it's definitely possible.
And of interest.
I did have one person approach me at
a meeting once asking
me if I could do flags.
He was a flag nut, so he wanted to know
if I could find the flags that
were made from the same cloth.
But the other thing I should mention
is that we are thinking, with Jonathan Hay
who's sitting right, Jon
you wanna raise your hand?
Who's at NYU, we're thinking about trying
to apply this to Chinese silk paintings.
Part of the reason that hasn't been done
is that the thread
count in silk is so high
that no human wants to bother.
But it turns out it has, the early ones
tend to be a tabby or a simple weave,
so our algorithm applies directly,
and you see stripes, and we should
be able to identify paintings that
come from the same bolt of silk.
Or at least we're crossing our fingers,
we believe that to be true at this point.
I didn't miss...
I got it right, okay.
Yes ma'am.
- [Audience Member] So
my question is along
the lines of all previous
questions about all the
medians or analog regardless
of canvas or paper or silk,
it all has the analog signature.
What happens to the digital arts?
Is there any, because after you imagine
you're done with all the physical objects,
what happens to all the digital art
that is not available for such methods?
- Okay, this one I'm gonna
have to say I don't know.
I haven't really given it enough thought.
I really can't answer your question.
I have no doubt that
ultimately there will,
I mean what I tend to do is I tend to
figure out what the art historian
is using to make these choices.
And then try to come up
with a tool that gives
them an enhanced ability
to make that choice.
So what that would means is I'd have to
spend time with someone who was trying
to make those kinds of choices with
the more modern media based art
and see what they're using to make these
choices of style and materials and such,
and then conceivably hope that
there would be some way to automate
the tedious part of what they're doing.
But right now I can't
say anymore than that.
Haven't thought about it.
Not enough time.
Retirement looms.
(laughter)
- [Stephen] Thank you all very much--
- I have one Stephen, I think.
- [Stephen] Okay.
- [Staff Member] One last question.
- [Audience Member] Yes, hi.
I was wondering if there's been
any concern about whether publishing
the x-rays opens up,
makes it easier for fakers
because you're giving
them a roadmap to making--
- Yes.
So there is some concern.
The one x-ray we believe
we will not be able
to show is the one from the Gardner Museum
of The Concert which was stolen.
They're very concerned about that.
My reaction however is that giving them
this kind of information
isn't gonna change much.
I don't see how the
forgers are going to make
the canvas have the weave
that we're looking at.
I know that forgers will go by,
a cheap painting from the right period,
scrape the painting off,
and then use a period piece
to paint on, that's been done many times.
But to actually find a piece of canvas
that matched the one of the artists
you were trying to match I think
is probably beyond imagination,
I don't think that'll happen.
I'm not too worried, but
some other people are.
- [Staff Member] I think
that's all we have time for.
- [Stephen] Well thank you (mumbles)
express all our appreciation.
(applause)
- Thanks.
- Thank you so much.
- How'd it go?
- Perfect.
- Whoops.
That was you not me right?
- That was my microphone.
