Hello there, welcome to Serverless
Days ANZ 2020 my name is
Will Velida. I'm a software
engineer and Microsoft Data
Platform MVP based in Auckland,
New Zealand. I'm really excited
to be able to talk to you all
today about building event
driven applications using Cosmos
DB and Azure Functions. Whether
you're watching in New Zealand
or Australia or happen to be
anywhere else in the world, I
hope they are all healthy and
safe during these strange times
and I hope you enjoy this
session. Now usually take this
opportunity to ask the room
whether or not they've used
Azure Functions or Azure Cosmos
DB before, what their experiences
are, are they using in
production or they just playing
around with it just so I can set
expectations for the session.
But since I can't see you all
today, I'm just going to start
by saying that this session
doesn't require you to be an
expert in anything I present
today. You might be considering
using Azure Functions. You might
be considering using Cosmos DB
in isolation. You might be
considering using them together.
You might have deep extensive
experience using both of them.
So today is all about me just
showing you how straightforward
it is to use Cosmos DB and
Azure Functions together to
build some event driven
applications. I will be doing
some code demos today and these
will be in C#, so if you
have some C# knowledge, great.
If you don't, I'm still hopeful
that you can learn something
today.
So today I'll start off by
giving you all an overview of
what Azure Cosmos DB is and what
we can use it for. Then I'll
dive into the details and how we
can use Azure Functions to work
of our data in Cosmos DB via the
different bindings available to
us. Then I'll talk a little bit
about how we can use dependency
injection in our Azure Functions
to work of our Cosmos DB clients
and why we might opt for this over
using the bindings. Then I'll go
through a quick demo showing you how
you can use dependency injection
within your Functions to work
with Cosmos DB and share it
amongst different Functions
within a Function application
and I will also show you a
function that uses a Cosmos DB
trigger to listen to changes
that occur inside our Cosmos DB
containers. After all that, I'll
show you some resources that you
can explore further in your own
time and will have some time at
the end to answer any questions
that you guys might have.
Now for those of you who've
never used Azure Cosmos DB
before Cosmos DB is Microsoft
globally distributed multi model
database service. Using Azure
Cosmos DB we can build
databases using a variety of
different data models. We can
build applications that use
document model data either using
the SQL or mongoDB API. We can
build applications that work
with columnar data using the
Cassandra API. If you're
building applications that use
key value data, we can use the
Table API. For those of you who
are very familiar with Azure
Table Storage, this is
essentially just a globally
distributed version of Azure
Table Storage. Or perhaps you're
using a developing applications
that use graph data, and for
that we can use the Gremlin
API. Essentially, Cosmos DB
provides us with the ability to
build highly responsive and
available applications
worldwide, ensuring that our
data is as close to our users as
possible. The benefits that come
with Cosmos DB are: turnkey
global distribution, which means
that Cosmos DB will replicate
your data to wherever your users
are on a region as close as
possible to your users where
they are. We can add or remove
any Azure Region to our Cosmos
DB account at anytime and
essentially Cosmos DB will
replicate your data to
all regions associated with your
Cosmos DB account while your
application continues to be
highly available. Cosmos DB
provides elastic scalability of
both throughput and storage
worldwide. You can scale from up
to thousands to millions of
requests per second and only pay
for the throughput that you need
to help deal with any spikes and
workload. There's guaranteed low
latency at the 99th percentile
worldwide, so this helps us
build highly responsive planet
scale applications and Cosmos DB
guarantees less than 10
millisecond latency's for both
index reads and writes at both
at the 99th percentile, which
helps us, which provides us with
really rapid queries and highly
responsive applications. With
Cosmos DB, there are precisely
defined multiple consistency
options, so traditionally we
would have to choose between two
consistency extremes, strong and
eventual. But what cosmos DB
provides is 5 well defined
levels of consistency for us to
choose from depending on the
application requirements. And
there's no scheme or index
management, with some caveats,
so right out of the box, Cosmos
DB automatically indexes all
data and is schema agnostic,
meaning that we don't have to
worry about application downtime
or managing schemas or indexes.
We can use Cosmos DB in a
variety of different scenarios.
So, for example, we might have
an IoT application that requires
us to read telemetry recordings
from devices all across the
globe. We might have a mobile
game that's used by gamers all
across the world and we need to
replicate their data as close to
them as possible for rapid
performance. We might have web
or retail applications with
customers all across the globe,
and we need to provide them with
ultra fast retailing shopping
experiences. Essentially, if we
have a NoSQL database service
that's ultra fast and
requires minimal management,
then Cosmos DB is a great
choice for our data
requirements.
Azure Functions provides us with
the ability to create scalable
units of work or precise piece
of logic that can be run on
demand without us having to
worry about provisioning or
managing our infrastructure. By
using Azure Functions with
Cosmos DB, we don't have to
create a full blown application
to respond to changes in our
Cosmos DB database. Instead we
can create small reusable
Functions for specific tasks.
We get instant access to all our
data, meaning that we are able
to constantly query, update and
add new items to our database
via Azure Functions. The data
that we handle can be schemeless
this handle anything approach
makes it fairly straightforward
to create a variety of different
Functions that output to Azure
Cosmos DB. We can benefit from
scalable throughput, so if we
have hundreds of thousands of
Functions querying and right
into the same container, we can
scale up our request units to
handle the load and all
Functions can work in parallel
using our associated request
units and our data is guaranteed
guaranteed to be consistent.
We can benefit from global
replication. They're event driven
as well so we can listen to a
change feed in Azure Cosmos DB
environment, meaning that we
don't have to create specific
listening logic, we just keep an
eye out for changes that we're
listening for. There are no
limits, so Azure Functions can
execute in parallel and the
service spins up as many
instances as we need and we we
just set the parameters on that
and it's generally good for just
performing very quick tasks, so
the service can spin up a new
instance of Functions whenever an
event fires and closes them as
soon as that Function completes
and we only pay for the time
that our Functions are running.
And there are different ways
that we can use Azure Functions
to interact with our Cosmos DB
databases. So for example, we
may want to execute a Function
whenever a document is created
or modified within a collection.
So here we can use a Cosmos DB
trigger to run our Functions. We
might want to do a quick read on
our collection based off a very
simple SQL query. So for this we
would use an input binding or we
might want to write to our
Cosmos DB collections and for
that we can use an output
bindings. Let's go through
some visual examples of
these bindings.
So let's start with the Cosmos
DB trigger. This is a really
simple way of using a really,
really cool feature in Azure
Cosmos DB called the change
feed. Essentially what we're
doing here is, we'll set up
our Function with the Cosmos
DB trigger to listen to a
container in our Cosmos DB
account, and every time a
document is created or
updated, our Azure Function
will be invoked and we can
produce an output from those
changes.
So let's go through visual
example here. Let's say we have
a function called purchase
product. We have an ecommerce
site and every time a customer
creates an order, we can use the
change feed to produce an output
depending on what we might want
to do. So, for example, we could
use the change feed to persist
our document to another
collection in Cosmos DB that
might have a different partition
key. We may want to take our
JSON document and store it into
a colder storage that's more
suitable for analytical or
archive workloads. Or perhaps if
you're familiar with Azure Data
bricks. We can use the change
feed connector for Spark listen
to the change feed and perform
some machine learning jobs based
off changes that occur in our
collections. So essentially, any
changes that occur within our
container in Cosmos DB is
produced as a sorted list of
documents in the order of which
they were modified, and they
can be processed
incrementally. Now with the
change feed, we aren't limited
to just a single consumer. It's
not like we have to set up
multiple change feeds in order
to do different things. All of
these consumers can consume the
same change feed to do parallel
processing. The change feed is
enabled by default on all Cosmos
DB accounts, so we don't have to
do anything special, there's no,
you know, elaborate
configuration. All we have to do
is set up our Function using the
Cosmos DB trigger and we can
start processing changes on our
Cosmos DB accounts, container,
sorry, right away. Now, Azure
Functions aren't the only way
that we can work with the change
feed in Cosmos DB, there is a
processor library, but that's
out of scope for this talk. So
that's just a real quick
introduction to what the change
feed in Cosmos DB is and how you
can use Azure Functions to work
with it. So let's go back to our
other bindings. We can use an
input binding to read data in
our Cosmos DB collections. So
for example, we might have an
Azure Function that has an HTTP
trigger as part of a serverless
API and we want to perform a
particular query that retrieves
data from our Cosmos DB
containers. We can then use that
output to return documents to
the user as part of a web page.
Essentially, if we need a subset
of our data in a Cosmos DB
collection, we can use an
input binding to execute really
simple queries to get that data
back to us.
Finally, we can use an output
binding to write data to
container in Cosmos DB. So let's
take our HTTP trigger, but this
time we're going to make a post
request as part of our serverless
API. Or perhaps we might be
reading events of Event Hub. We
would want to consume those
events within a Function and
then persist it outputs to
Cosmos DB. We can set up an
Azure Function with Cosmos DB
output binding to write that
data to our containers.
Now bindings are really great to
quickly set things up and
execute some very simple,
straightforward tasks with
Cosmos DB. But there are some
limitations. Currently the
bindings only support the core
or the SQL API, so like I said
at the start there are five APIs
for us to choose from for our
Cosmos DB accounts, but if our
cosmos DB account wasn't
provisioned using the SQL API
and we want to interact with
that Cosmos DB account using our
Azure Functions, we would have
to use a static client in order
to do that. Now each binding
also instantiates its own
client, so if we have a function
that just executes
every so often, this isn't a
problem. But say we have
multiple Functions trying to use
our Cosmos database. This would
cause a massive performance hit
because all of these different
Functions would instantiate
their own version of the client.
And the bindings themselves
still use version 2 of the .NET
SDK. Now if you have a bit of
experience using Cosmos DB and
you've been using it in C#,
you would have known that
version 3 of the SDK's been
out for quite a while and
there's been some massive
improvements in terms of
performance and ease of use for
the SDK. So if you wanted to
continue to use that latest and
greatest version of the SDK, you
couldn't use the bindings 'cause
they still use version 2.
However. With Azure, with Azure
Functions from version two,
there's support for dependency
injection. These are similar to
the .NET Core dependency
injection features and they have
the same service lifetimes,
as ASP.NET dependency injection,
so that's Transient, Scoped and
Singleton service
lifetimes that we can create for
our Cosmos DB clients.
It's now time for our demo, now
for this demo what we're going
to do is we're going to set up a
Function that sends events to an
Event Hub via an HTTP trigger,
and then have another Function
that listens to those events on
that Event Hub and persist the
changes to a Cosmos DB container
via Singleton instance of our
Cosmos DB client. We're then
going to listen to changes in
our Cosmos DB container using
the change feed, fire Cosmos DB
trigger function that will
produce a materialized view of our
data in another Cosmos
DB collection.
Alright, so here we are in
Visual Studio. Now what I've
done is I've gone ahead and
created our Function App just so
you don't have to sit and watch
me code, but we're going to take
this step by step. So let's
start with the data that we're
going to be using. I've created
a small sample device reading
class for us to just generate
some synthetic data. Now, as you
may have noticed, I'm decorating
my class property names with
some JsonProperty attributes
and these will just be the paths
that we see in our Cosmos DB
documents. Moving onto our
startup class. So in this class
we'll be using dependency
injection to inject the services
that will be using throughout
our Function, Function App. I
start by creating a method here
called Configure. Let me just
create some real estate in
Visual Studio and I'll increase
the size. Sweet. So I've got this
configure method here and I'm
adding components to an
IFunctionsHostBuilder and in
this method I'm adding some
logging, adding some simple
configuration that I will just
be loading from a local settings
JSON file. Before adding my
Singleton instances of both my
Cosmos DB client and event
processor client. As you can see
here, I'm creating some Cosmos
client options and we're just
essentially saying that we can
in this setup class we can
configure our clients in one
place and then just reuse that
throughout our different Functions.
And now because I've set up my
Singleton services, I'll be able
to inject them into my Functions
as and when I need them.
Cool, so let's take a look at
our Functions. Now, in our first
Functions, Function, sorry, called
TriggerEvents all we're doing
here is we've got a simple HTTP
trigger that's just going to
send a post request, and on that
post request we're just going to
create some fake device reading
events. I'm just using a library
called bogus to generate some
fake device readings. And that
will just create a list and then
for each reading within that
list, we're going to send it to
Event Hub here in these two
lines. So here I am injecting my
logging service and Event Hub
producer clients into this
Function and then here we are
sending our event to Event Hub.
In our second Function
PersistRawReadingsToCosmos
we're tracking, triggering this
Function, sorry, based off the
events that are being sent to
our Event Hub. So for each
event that we receive in our
array of events we are
creating a new item
that we are going to persist into
Cosmos DB, into our Cosmos DB
container, and if we're
successful, we're just going to
log the device ID to the
console, just saying that the
item is being persisted. So
here I've got some static
versions of my logging,
configuration and my Cosmos client
and then just using constructor
injection to inject those
services into this Function.
Fairly straightforward. Now in
our final Function
PersistMaterializedReadingsToCosmosDB. 
Here, we're using the
Cosmos DB trigger
and this acts as a change feed
listener on our container that
we're persisting items into the
container that we're persisting
items to based on our previous
Function. So we'll have two
collections and I'll show you
this in the portal in the Azure
portal later, but essentially
what's happening here is that
the change feed will
periodically poll our container
for changes before performing
actions onto it. Now by default
this Function will poll the
change feed every 5 seconds, but
we can actually set this time
interval ourselves by using the
FeedPollDelay attribute
within our trigger so I could
set it to 30 seconds and then
essentially what will happen is
that this trigger, this Function,
sorry, will be invoked every 30
seconds and then it will poll
for the container that it's
listening to for any changes
that have occurred in that
container before performing any
actions. But we'll just
leave the default for
five seconds for now.
And all the changes that occur
within this time frame will get
added to this IReadOnlyList
here of type document
which we then iterate over to
perform actions of those, on
items, within that list. So here
for example, for here in our
Function, what we're doing is
that for every document within
our list we are creating a new
document in our collection
within Cosmos DB. Now the
difference here is that in our
raw readings container we're
partitioning on the Device ID
and for materialized
readings were partitioning on
our device location. So if
we're performing queries, we
can search for readings
based on location rather
than the ID, and that's
a more read friendly approach
to Cosmos DB.
OK, so let's start our Function
and what will should happen is
once our Functions have started
up, we'll get a URL which we can
send a POST request to, which
will kick off, triggering our,
sending our events to Event Hub.
So let's do that. Now go into
Postman. There's our URL for our
Function. We're just going to be
sending a POST request, so if I
press send. Open back, open up
the Function console and we can
see events here being sent to
Event Hub and those events being
persisted into Azure Cosmos DB.
And we can see some of those
events being materialized.
So yeah, we can see device
readings from Europe. Some
device readings from NZ being
materialized if we go into Azure
Cosmos DB. What we can see here
is I've got a few collections.
So here are my collections for
my raw readings. So these
collections are, well these
documents inside
this collection of the raw event
from event that we received from
Event Hub ourselves. So we've
got it partitioned on ID. And then
I've got another collection here
for materialized readings. So
let's open that up.
As you can see, they're partitioned
by location. So say if I wanted
to, this just makes it a little
bit easier to perform simple
queries against for more
materialized view. So c.location
is the property and I'm
just going to search for the
ones in UAE, apply the filter.
And there we go. So there are all
the device readings that we've
recorded from devices in the
United Arab Emirates. So that's
a real simple way of basically
tying some Functions together.
Reading some events from Event
Hub that we've sent to Event
Hub, persisting into Azure
Cosmos DB and then taking those
documents within one container
in Cosmos DB using the change
feed to listen to changes
occurring inside that container
and then adding them, sorry,
persisting them to this
materialize readings collection.
Now you may have notice this
leases collection as well, so if
we just look in here this add
this acts, sorry, as a checkpoint
for our change feed. So
essentially the change feed will
poll the container every 30
seconds once it's processed all
the changes within that first
second period it will add a
checkpoint so when it polls it
again it will know where to
listen, the change feed will know
where to listen from next.
So that wraps it up for our
demo. Now if you want to try
this stuff out or you want to
learn about Azure Functions for
free, check out those links
right there. If you want to keep
up to date with what's happening
in the community, make sure you
subscribe to the newsletter and
I believe the ANZ Serverless Days
team have organized some free
swag, so if you want to claim
that, just check out that link
right there. For the New Zealand
based developer audiences, make
sure you join the community for
regular updates from community
experts and also invites to
community events and workshops.
And if you're looking to
learn more stuff in general,
make sure you check out
Microsoft Learn. There's some
fantastic free courses and
tutorials there, not just on
Azure Functions and Cosmos
DB, but you from a variety of
different technologies.
Perhaps you're preparing for
certifications, or you're
just want to learn something
new, I'd highly recommend
checking out Microsoft Learn
to upskill yourself.
So that's me for this
presentation. If you have
any questions, there will
be a bit of a Q&A after
this presentation, but if
you think of any other
questions after that, feel
free to connect with me on
Twitter. I'm fairly
responsive on there, and if
you want to see the code
that I demonstrated for
this presentation today,
please check out my GitHub.
So thanks for joining us. Are
there any questions?
