Cosmos DB Server-Side Programming with TypeScript – Part 3: Stored Procedures

Stored procedures, the second type of server-side code that can run within Cosmos DB, provide the ability to execute blocks of functionality from inside the database engine. Typically we use stored procedures for discrete tasks that can be encapsulated within a single invocation. In this post, we will discuss some situations where stored procedures can be used and the actions and queries that they can perform. We’ll then start to work through the server-side API model, and look at how we can work with the incoming stored procedure invocation’s request and response as well as the Cosmos DB collection itself. Then we’ll build a simple stored procedure using TypeScript.

This post is part of a series of posts about server-side programming for Cosmos DB:

  • Part 1 gives an overview of the server side programmability model, the reasons why you might want to consider server-side code in Cosmos DB, and some key things to watch out for.
  • Part 2 deals with user-defined functions, the simplest type of server-side programming, which allow for adding simple computation to queries.
  • Part 3 (this post) talks about stored procedures. These provide a lot of powerful features for creating, modifying, deleting, and querying across documents – including in a transactional way.
  • Part 4 (coming soon) introduces triggers. Triggers come in two types – pre-triggers and post-triggers – and allow for behaviour like validating and modifying documents as they are inserted or updated, and creating secondary effects as a result of changes to documents in a collection.
  • Part 5 (coming soon) discusses unit testing your server-side scripts. Unit testing is a key part of building a production-grade application, and even though some of your code runs inside Cosmos DB, your business logic can still be tested.
  • Finally, part 6 (coming soon) explains how server-side scripts can be built and deployed into a Cosmos DB collection within an automated build and release pipeline, using Microsoft Visual Studio Team Services (VSTS).

Stored Procedures

Stored procedures let us encapsulate blocks of functionality, and then later invoke them with a single call. As with the other server-side code types, the code inside the stored procedure run inside the Cosmos DB database engine. Within a stored procedure we can perform a range of different actions, including querying documents as well as creating, updating, and deleting documents. These actions are done within the collection that the stored procedure is installed in. Of course, we can augment these collection-based actions with custom logic that we write ourselves in JavaScript or – as is the case in this series – TypeScript.

Stored procedures are extremely versatile, and can be used for a number of different tasks including:

  • Encapsulating a complex set of queries and executing them as one logical operation – we will work with this example below.
  • Retrieving data for a report using a complex set of queries, and combining the results into a single response that can be bound to a report UI.
  • Generating mock data and inserting it into the collection.
  • Doing a batch insert, update, upsert, or delete of multiple documents, taking advantage of the transactional processing of stored procedures.

Of course, if you are building a simple application without the need for complex queries, you may be able to achieve everything you need with just the Cosmos DB client-side SDKs. However, stored procedures do give us some power and flexibility that is not possible with purely client-based development, including transaction semantics.

Transactions and Error Handling

Cosmos DB’s client programming model does not provide for transactional consistency. However, stored procedures do run within an implicit transaction. This means that if you modify multiple documents within a stored procedure, then either all of those changes will be saved or – in the event of an error – none of them will be saved. Transactions provide four guarantees (atomicity, consistency, isolation, and durability, also known as ACID). More information on Cosmos DB transactions is available here.

The Cosmos DB engine handles committing and rolling back transactions automatically. If a stored procedure completes without any errors being thrown then the transaction will be committed. However, if even one unhandled error is thrown, the transaction will be rolled back and no changes will be made.

Working with the Context

Unlike user-defined functions, which we discussed in part 1 of this series, stored procedures allow us to access and make changes to the collection that they run within. We can also return results from stored procedures. Both of these types of actions require that we work with the context object.

Within a stored procedure, we can make a call to the getContext() function. This returns back an object with three functions.

  • getContext().getRequest() is used to access the request details. This is mostly helpful for triggers, and we will discuss this in part 4 of this series.
  • getContext().getResponse() lets us set the response that we should send back. For stored procedures, this is the way that we will return data back to the client if the stored procedure has something to return.
  • getContext().getCollection() gives us access to the Cosmos DB collection that the stored procedure runs within. In turn, this will let us read and write documents.

Each of the calls above corresponds to a type – Context, Request, Response, and Collection, respectively. Each of these types, in turn, provide a set of functions for interacting with the object. For example, getContext().getCollection().queryDocuments() lets us run a query against documents in the collection, and getContext().getResponse().setBody() lets us specify the output that we want the stored procedure to return. We’ll see more of these functions as we go through this series.

Also note that the double-underscore (__) is automatically mapped to the getContext().getCollection() function. In this series we won’t use this shortcut because I want to be more explicit, especially to help with testing when we get to part 5.

Type Definitions

Human-readable documentation for the types and their members is provided by Microsoft here. Of course, one of the main reasons we’re using TypeScript in this series is so that we get type checking and strong typing against the Cosmos DB object model, so a human-readable description isn’t really sufficient. In TypeScript, this is done through type definitions – descriptions of types and their members that TypeScript can use to power its type system.

While Microsoft doesn’t provide first-party TypeScript type definitions for Cosmos DB, an open-source project named DefinitelyTyped provides and publishes these definitions here. (Note that the type definitions use the old name for Cosmos DB – DocumentDB – but they are still valid and being updated.)

Queries

One of the main things we’ll frequently do from inside stored procedures is execute queries against Cosmos DB. This is how we can retrieve data and perform custom logic on it within the stored procedure. Cosmos DB provides an integrated JavaScript query syntax for executing queries. The syntax is documented here and lets us write queries like this:

 

 

 

 

 

 

 

 

 

which will map to the following SQL query:

 

 

 

 

However, there are some limitations to this query syntax. We can’t perform aggregations, and we can’t do queries using user-defined functions. These limitations may be lifted in future, but for now, in this series we will use the SQL syntax instead so that we can get the full power of Cosmos DB’s query engine. We can use this type of syntax to make a SQL-based query:

 

 

 

 

 

 

 

 

In your own stored procedures and triggers you can decide which approach – integrated JavaScriptor SQL – makes more sense.

We can also request a document by ID if we want to as well:

 

 

 

 

 

 

Another important consideration when executing queries is that Cosmos DB limits the amount of time that a stored procedure can run for. We can test whether our stored procedure is approaching that time limit by inspecting the return value we receive when we submit the query. If we receive a false response, it means the query wasn’t accepted – and that it’s probably time to wrap up our logic before we get forcibly shut off. In some stored procedures, receiving a false like this may mean that you simply throw an error and consider the whole stored procedure execution to have failed.

Parameterised Queries

In the last section, we discussed using the collection.queryDocuments function to execute a SQL query against the collection. While this technically works, once we start including parameters in our queries then we’d need to concatenate them into the query string. This is a very, very bad idea – it opens us up to a class of security vulnerabilities called SQL injection attacks.

When we’re writing SQL queries with parameters, we should instead the overload of the collection.queryDocuments function that accepts an IParameterizedQuery instead of a string. By doing this, we pass our parameters explicitly and ensure that they are handled and escaped appropriately by the database engine. Here’s an example of how we can do this from our TypeScript code:

 

 

 

 

 

Updating Documents

We can also make changes to documents within the collection,. There are several functions on the Collection type to help with this, including:

  • createDocument inserts a new document into the collection.
  • replaceDocument updates an existing document in the collection. You must provide the document link to use this function.
  • deleteDocument deletes a document from the collection.
  • upsertDocumentThere are also functions to deal with attachments to documents, but we won’t work with those in this series.

These functions that work with existing documents also take an optional parameter to specify the etag of the document. This allows for us to take advantage of optimistic concurrency. Optimistic concurrency is very useful, but is outside the scope of this series.

Structuring Your Stored Procedure Code

Stored procedures will often become more complex than UDFs, and may incorporate business logic as well as interaction with the Cosmos DB collection inside the code. When we’re writing a production-grade application it is important that we structure our code correctly so that each part is testable, and has a clearly defined responsibility and interactions with other components. In Cosmos DB, we are interacting with the collection in order to execute queries and update documents, and these side effects are important to test as well. We’ll discuss testing in more detail in part 5, but even before we worry about testing, it’s a good idea to structure our code properly.

When I write Cosmos DB stored procedures, I find it’s helpful to have a simple ‘entry point’ function. This entry point does the interaction with the getContext() function and retrieves the Collection, Request, and Response objects as required. These, along with any other parameters, are then passed into an internal implementation function, which in turn may invoke other functions to do other parts of the logic. By structuring the functions in this way we can ensure that each function has a clear purpose, and that the external components can be mocked and/or spied upon during our tests.

Writing our stored procedure in TypeScript also gives us the ability to store our functions in different .ts files if we want. This is helpful when we have long and complicated stored procedures, or if we want to keep interfaces and functions in separate files. This is largely a choice of style, since TypeScript’s compiler will simply combine all of the functions together at compilation time and will emit a single JavaScript output file (because we have set the outFile property in our tsconfig.json file). One important note on this though – if you have functions spread across multiple files, it is important to pay attention to the order in which the functions get emitted. The stored procedure’s ‘entry point’ function must appear first in the output JavaScript file. TypeScript can be instructed to do this by explicitly listing the entry point function’s file first in the include directive within the tsconfig.json file, and then having a wildcard * to catch the remaining files, like this:

 

 

 

 

 

 

 

Calling Stored Procedures

Once a stored procedure is written, we need to call it in order to check that it’s working, and then to use it in our real applications. There are several ways we can call our stored procedure and pass in the arguments it expects.

  • The Azure Portal provides a test interface for invoking stored procedures. This is how we will test the stored procedure we write below.
  • The client SDKs provide platform-specific features for invoking stored procedures. For example, the .NET client library for Cosmos DB provides the DocumentClient.ExecuteStoredProcedureAsync function, which accepts the ID of a stored procedure and any arguments that it might be expecting.
  • The Cosmos DB REST API also allows for invoking stored procedures directly.

Again, the exact way you call stored procedures may depend on the Cosmos DB API you are targeting – in this series we are using the SQL API, and the invocation mechanisms for MongoDB, Gremlin, and the other APIs may be different.

Now that we have talked about the different aspects of writing stored procedures, we can think about how we might use a stored procedure in our sample scenario.

Defining our Stored Procedure

In our hypothetical ordering application, we have a Cosmos DB collection of documents representing orders. Each order document contains a customer’s ID (as we saw in part 2 of this series), and also contains a set of items that the customer ordered. An order item consists of the product ID the customer ordered and the quantity of that product.

Because Cosmos DB is a schemaless database, our order documents may coexist with many other documents of different types. Therefore, we also include a type parameter on the document to indicate that it is an order. This type discriminator pattern is quite common in schemaless databases.

An example of an order document in our collection is:

For this stored procedure we want to pass in a set of product IDs, and get back a grouped list of IDs for customers who have ordered any of those products. This is similar to doing a GROUP BY in a relational database – but currently Cosmos DB doesn’t provide this sort of grouping feature, and so we are using a stored procedure to fill in the gap. Doing this from the client SDK would require multiple queries against the collection, but by using a stored procedure we can just make one call.

At a high level, our stored procedure logic looks like this:

  1. Accept a list of product IDs as an argument.
  2. Iterate through the product IDs.
  3. For each product ID, run a query to retrieve the customer IDs for the customers that have ordered that product, filtering to only query on order documents. The query we’ll run looks like this:

 

 

  1. Once we have all of the customer IDs for each product in our list, create a JSON object to represent the results like this:

Preparing a Folder

Now we can start writing our stored procedure. If you want to compare against my completed stored procedure, you can access it on GitHub.

In part 2 of this series we covered how to set up the Cosmos DB account and collection, so I won’t go through that again. We also will reuse the same folder structure as we did in part 2, so you can refer to that post if you’re following along.

There’s one major difference this time though. In our package.json file, we need to add a second entry into the devDependencies list to tell NPM that we want to include the TypeScript type definitions for Cosmos DB. Our package.json file will look like this:

 

 

 

Open a command prompt or terminal window, and run npm install within this folder. This will initialise TypeScript and the type definitions.

We’ll also adjust the tsconfig.json file to emit a file with the name sp-getGroupedOrders.js:

Writing the Stored Procedure

Now let’s create a file named src/getGroupedOrders.ts. This is going to contain our stored procedure code. Let’s start with adding a basic function that will act as our entry point:

As discussed above, this is a pattern I try to follow when I write Cosmos DB server-side code. It helps to keep the interaction with the getContext() function isolated to this one place, and then all subsequent functions will work with explicit objects that we’ll pass around. This will help when we come to test this code later. You can see that this function calls the getGroupedOrdersImpl function, which does the actual work we need done – we’ll write this shortly.

Before then, though, let’s write a basic interface that will represent our response objects:

 

Our getGroupedOrdersImpl function will accept an array of product IDs and the collection in which to query them, and it will return an array of these CustomersGroupedByProducts. Of course, since CustomersGroupedByProduct is a TypeScript interface, we can use it within our functions for type safety, but it will be stripped out when we compile the code into JavaScript.

Now we can add a basic shell of an implementation for our getGroupedOrdersImpl function. As you type this, notice that (if you’re in an editor that supports it, like Visual Studio Code) you get IntelliSense and statement completion thanks to the TypeScript definitions:

This function prepares a variable called outputArray, which will contain all of our product/customer groupings. Then we have some placeholder code to perform our actual queries, which we’ll fill in shortly. Finally, this function returns the output array.

Now we can fill in the placeholder code. Where we have REPLACEME in the function, replace it with this:

There’s a lot going on here, so let’s break it down:

  • The first part (lines 1-6) sets up a new IParameterizedQuery, which lets us execute a SQL query using parameters. As discussed above, this is a much more secure way to handle parameters than string concatenation. The query will find all orders containing the product ID we’re looking for, and will return back the customer ID.
  • Next, the query callback function is prepared (lines 7-18). This is what will be called when the query results are available. In this function we pull out the results and push them onto our outputArray, ready to return to the calling function.
  • Then we try to execute the query against the collection by using the collection.queryDocuments() function (line 19). This function returns a boolean to indicate whether the query was accepted (line 20). If it wasn’t, we consider this to be an error and immediately throw an error ourselves (line 22).

That’s it! The full stored procedure file looks like this:

Here’s what the your folder structure should now look like:

  • /
    • package.json
    • tsconfig.json
    • src/
      • getGroupedOrders.ts

Compiling the Stored Procedure

As in part 2, we can now compile our function to JavaScript. Open up a command prompt or terminal, and enter npm run build. You should see that a new output folder has been created, and inside that is a file named sp-getGroupedOrders.js. If you open this, you’ll see the JavaScript version of our function, ready to submit to Cosmos DB. This has all of the type information removed, but the core logic remains the same. Here’s what it should look like:

Deploying the Stored Procedure

Now let’s deploy the stored procedure to our Cosmos DB collection. Open the Azure Portal, browse to the Cosmos DB account, and then go to Script Explorer. Click Create Stored Procedure.

cosmos-sp-1

Enter getGroupedOrders as the ID, and then paste the contents of the compiled sp-getGroupedOrder.js JavaScript file into the body.

cosmos-sp-2

Click Save to install the stored procedure to Cosmos DB. (Once again, this isn’t the best way to install a stored procedure – we’ll look at better ways in part 6 of this series.)

Testing the Stored Procedure

Now let’s insert some sample data so that we can try the stored procedure out. Let’s insert these sample documents using Document Explorer, as described in part 2.

Here are the three sample documents we’ll use:

Now go back into Script Explorer, open the stored procedure, and notice that there is a test panel below the script body textbox. We can enter our stored procedures parameters into the Inputs field. Let’s do that now. Enter [["P1", "P2", "P10"]] – be careful to include the double square brackets around the array. Then click the Save & Execute button, and you should see the results.

cosmos-sp-3

If we reformat them, our results look like the following. We can see that we have an array containing an object for each product ID we passed into the query, and each object has a list of customer IDs who ordered that product:

So our stored procedure works! We’ve now successfully encapsulated the logic involved in querying for customers that have ordered any of a set of products.

Summary

Stored procedures give us a way to encapsulate queries and operations to be performed on a Cosmos DB collection, and to invoke them as a single unit. Stored procedures run within an implicit transaction, so any unhandled errors will result in the changes being rolled back. Unlike in UDFs, we are also able to access the collection within a stored procedure by using the getContext() function, and by retrieving the Response and Collection objects. This allows us to return rich data, including objects and arrays, as well as to interact with the collection and its documents. In the next part of this series we will discuss triggers, the third type of server-side programming available in Cosmos DB, which allow us to intercept changes happening to documents within the collection.

Key Takeaways

  • Stored procedures encapsulate logic, queries, and actions upon documents within the collection.
  • Stored procedures provide transactional isolation, and all stored procedures run within a transaction scope.
  • The getContext() function allows us to access the Response and Collection objects.
  • TypeScript definitions are available to help when writing code against these objects.
  • Cosmos DB provides an integrated query syntax, which is great for simple queries, but doesn’t cover all possible queries that can be executed against the collection.
  • Arbitrary SQL queries can also be executed. If these contain parameters then the IParameterizedQuery interface should be used to ensure safe coding practices are adhered to.
  • The order of functions inside the stored procedure’s file matters. The first function will be the one that Cosmos DB treats as the entry point.
  • You can view the code for this post on GitHub.

Cosmos DB Server-Side Programming with TypeScript – Part 2: User-Defined Functions

User-defined functions (UDFs) in Cosmos DB allow for simple calculations and computations to be performed on values, entities, and documents. In this post I will introduce UDFs, and then provide detailed steps to set up a basic UDF written in TypeScript. Many of these same steps will be applicable to stored procedures and triggers, which we’ll look at in future posts.

This is the second part of a series of blog posts on server-side development using Cosmos DB with TypeScript.

  • Part 1 gives an overview of the server side programmability model, the reasons why you might want to consider server-side code in Cosmos DB, and some key things to watch out for.
  • Part 2 (this post) deals with user-defined functions, the simplest type of server-side programming, which allow for adding simple computation to queries.
  • Part 3 talks about stored procedures. These provide a lot of powerful features for creating, modifying, deleting, and querying across documents – including in a transactional way.
  • Part 4 (coming soon) introduces triggers. Triggers come in two types – pre-triggers and post-triggers – and allow for behaviour like validating and modifying documents as they are inserted or updated, and creating secondary effects as a result of changes to documents in a collection.
  • Part 5 (coming soon) discusses unit testing your server-side scripts. Unit testing is a key part of building a production-grade application, and even though some of your code runs inside Cosmos DB, your business logic can still be tested.
  • Finally, part 6 (coming soon) explains how server-side scripts can be built and deployed into a Cosmos DB collection within an automated build and release pipeline, using Microsoft Visual Studio Team Services (VSTS).

User-Defined Functions

UDFs are the simplest type of server-side development available for Cosmos DB. UDFs generally accept one or more parameters and return a value. They cannot access Cosmos DB’s internal resources, and cannot read or write documents from the collection, so they are really only intended for simple types of computation. They can be used within queries, including in the SELECT and WHERE clauses.

UDFs are simple enough that types are almost not necessary, but for consistency we will use TypeScript for these too. This will also allow us to work through the setup of a TypeScript project, which we’ll reuse for the next parts of this series.

One note on terminology: the word function can get somewhat overloaded here, since it can refer to the Cosmos DB concept of a UDF, or to the TypeScript and JavaScript concept of a function. This can get confusing, especially since a UDF can contain multiple JavaScript functions within its definition. For consistency I will use UDF when I’m referring to the Cosmos DB concept, and function when referring to the JavaScript or TypeScript concept.

Parameters

UDFs can accept zero or more parameters. Realistically, though, most UDFs will accept at least one parameter, since UDFs almost always operate on a piece of data of some kind. The UDF parameters can be of any type, and since we are running within Cosmos DB, they will likely be either a primitive type (e.g. a single string, number, array, etc), a complex type (e.g. a custom JavaScript object, itself comprised of primitive types and other complex types), or even an entire document (which is really just a complex type). This gives us a lot of flexibility. We can have UDFs that do all sorts of things, including:

  • Accept a single string as a parameter. Do some string parsing on it, then return the parsed value.
  • Accept a single string as well as another parameter. Based on the value of the second parameter, change the parsing behaviour, then return the parsed value.
  • Accept an array of values as a parameter. Combine the values using some custom logic that you define, then return the combined value.
  • Accept no parameters. Return a piece of custom data based on the current date and time.
  • Accept a complex type as a parameter. Do some parsing of the document and then return a single output.

Invoking a UDF

A UDF can be invoked from within the SELECT and WHERE clauses of a SQL query. To invoke a UDF, you need to include the prefix udf. before the function name, like this:

SELECT udf.parseString(c.stringField) FROM c

In this example, udf. is a prefix indicating that we want to call a UDF, and parseString is the name of the UDF we want to call. Note that this is identifier that Cosmos DB uses for the UDF, and is not necessarily the name of the JavaScript function that implements the UDF. (However, I strongly recommend that you keep these consistent, and will do so throughout this series.)

You can pass in multiple parameters to the UDF by comma delimiting them, like this:

SELECT udf.parseString(c.stringField, 1234) FROM c
SELECT udf.parseString(c.stringField1, c.stringField2) FROM C

To pass a hard-coded array into a UDF, you can simply use square brackets, like this:

SELECT udf.parseArray(["arrayValue1", "arrayValue2", "arrayValue3"])

Now that we’ve talked through some of the key things to know about UDFs let’s try writing one, using our sample scenario from part 1.

Defining our UDF

Let’s imagine that our order system was built several years ago, and our business has now evolved significantly. As a result, we are in the middle of changing our order schema to represent customer IDs in different ways. Cosmos DB makes this easy by not enforcing a schema, so we can simply switch to the new schema when we’re ready.

Our old way of representing a customer ID was like this:

Now, though, we are representing customers with additional metadata, like this:

However, we still want to be able to easily use a customer’s ID within our queries. We need a way to dynamically figure out the customer’s ID for an order, and this needs to work across our old and new schemas. This is a great candidate for a UDF. Let’s deploy a Cosmos DB account and set up this UDF.

Setting Up a Cosmos DB Account and Collection

First, we’ll deploy a Cosmos DB account and set up a database and collection using the Azure Portal. (Later in this series, we will discuss how this can be made more automatable.) Log into the Azure Portal and click New, then choose Cosmos DB. Click Create.

cosmos-udf-1

We need to specify a globally unique name for our Cosmos DB account – I have used johnorders, but you can use whatever you want. Make sure to select the SQL option in the API drop-down list. You can specify any subscription, resource group, and location that you want. Click Create, and Cosmos DB will provision the account – this takes around 5-10 minutes.

Once the account is created, open it in the Portal. Click Add Collection to add a new collection.

cosmos-udf-2

Let’s name the collection Orders, and it can go into a database also named Orders. Provision it with a fixed (10GB) capacity, and 400 RU/s throughput (the minimum).

cosmos-udf-3

Note that this collection will cost money to run, so you’ll want to remember to delete the collection when you’re finished. You can leave an empty Cosmo DB account for no charge, though.

Preparing a Folder

TypeScript requires that we provide it with some instructions on how to compile our code. We’ll also use Node Package Manager (NPM) to tie all of our steps together, and so need to prepare a few things before we can write our UDF’s code.

Note that in parts 2, 3, and 4 of this series, we will write each server-side component as if it was its own independent application. This is to keep each of these posts easy to follow in a standalone way. However, in parts 5 and 6, we will combine these into a single folder structure. This will more accurately represent a real-world application, in which you are likely to have more than one server-side component.

Create a new folder on your local machine and add a file named package.json into it. This contains the NPM configuration we need. The contents of the file should be as follows:

The project.json file does the following:

  • It defines the package dependencies we have when we develop our code. Currently we only have one – typescript.
  • It also defines a script that we can execute from within NPM. Currently we only have one – build, which will build our UDF into JavaScript using TypeScript’s tsc command.

At a command prompt, open the folder we’ve been working in. Run npm install, which will find and install the TypeScript compiler. If you don’t have NPM installed, install it by following the instructions here.

Next, create a file named tsconfig.json. This is the TypeScript project configuration. This should contain the following:

The tsconfig.json instructs TypeScript to do the following:

  • Find and compile all of the files with the .ts extension inside the src folder (we haven’t created this yet!).
  • Target ECMAScript 2015, which is the version of JavaScript that Cosmos DB supports. If we use more modern features of TypeScript, it will handle the details of how to emit these as ECMAScript 2015-compatible code.
  • Save the output JavaScript to a single file named output/udf-getCustomerId.js. This filename is arbitrary and it could be any name we want, but I find it helpful to use the convention of  {kind}-{objectName}.js, especially as we add more code in later parts of this series.
    Note that the outFile directive means that, even if we included multiple source TypeScript files, TypeScript will save the complete compiled script into a single output file. This is in keeping with the requirement that Cosmos DB imposes that a server-side component has to be specified in a single file.

Writing the UDF

Now we can write our actual UDF code! If you want to compare against my completed UDF, you can access it on GitHub.

Create a folder named src, and within that, create a file named getCustomerId.ts. (Once again, this file doesn’t need to be named this way, but I find it helpful to use the UDF’s name for the filename.) Here’s the contents of this file:

Briefly, here’s an explanation of what this file does:
  • It declares a function named getCustomerId, which accepts a single parameter of type OrderDocument and returns a string. This is the function that represents our UDF.
  • The function inspects the document provided, and depending on which version of the schema it follows, it pulls the customer ID out of the appropriate field.
  • If the customer ID isn’t in either of the places it expects to find them, it throws an error. This will be further thrown up to the client by Cosmos DB.
  • Finally, it declares an interface named OrderDocument. This represents the shape of the data we’re expecting to store in our collection, and it has both of the ways of representing customer IDs.
    Note that we are using an interface and not a class, because this data type has no meaning to Cosmos DB – it’s only for use at development and build time.
    Also note that we could put this into its own orderDocument.ts file if wanted to keep things separated out.

At the end of this, your folder should look something like this:

  • /
    • package.json
    • tsconfig.json
    •  src/
      • getCustomerId.ts

You can access a copy of this as a GitHub repository here.

We have now written our first UDF! We’re almost ready to run it – but before then, we need to compile it.

Compiling the UDF

At the command line run npm run build. This will run the build script we defined inside the package.json file, which in turn simply runs the tsc (TypeScript compiler) command-line application. tsc will find the tsconfig.json file and knows what to do with it.

Once it’s finished, you should see a new output folder containing a file named udf-getCustomerId.js. This is our fully compiled UDF! It should look like the following:

If you compare this to the code we wrote in TypeScript, you’ll see that it is almost the same – except all of the type information (variable types and the interface) have been stripped away. This means that we get the type safety benefits of TypeScript at authoring and compilation time, but the file we provide to Cosmos DB is just a regular JavaScript file.

Deploying the UDF

Now we can deploy the UDF. Back in the Azure Portal, open Script Explorer under the Collections section, and then click Create User Defined Function.

cosmos-udf-4.png

Enter getCustomerId in the ID field. This will be the name we address the UDF by when we start to call it from our queries. Note that you don’t have to use the same ID field here as the JavaScript function name – in fact, the JavaScript function can be named anything you like. For clarify, though, I find it helpful to keep everything in sync.

Now we can copy and paste the contents of the udf-getCustomerId.js file into the large script text box.

cosmos-udf-5

Click Save to install the UDF to Cosmos DB.

Testing the UDF

Finally, let’s test the UDF! We’ll need to add a couple of pieces of sample data. Click Document Explorer under the Collections section, and then click the Create button. Paste in this sample document:

Click Save, and then close the blade and create a second document with the following contents:

This gives us enough to test with. Now click Query Explorer under the Collections section. Enter the following query:

SELECT c.id, udf.getCustomerId(c) AS customerId FROM c

This query does the following:

  • Refers to each document within the current collection (c).
  • Runs the getCustomerId UDF, passing in the document contents. Note that to refer to a UDF, you must prefix the name of the UDF with udf..
  • Projects out the document ID (id) and the customerId as customerId.

You should see the following output:

That’s exactly what we wanted to see – the UDF has pulled out the correct field for each document.

As a point of interest, notice the Request Charge on the query results. Try running the query a few times, and you should see that it fluctuates a little – but is generally around 3.5 RUs.

Now let’s try passing in an invalid input into our UDF. Run this query:

SELECT udf.getCustomerId('123') FROM c

Cosmos DB will give you back the error that our UDF threw because the input data (123) didn’t match either of the schemas it expected:

Encountered exception while executing Javascript.
  Exception = Error: Document with id undefined does not contain customer ID in recognised format.
  Stack trace: Error: Document with id undefined does not contain customer ID in recognised format.
    at getCustomerId (getCustomerId.js:11:5)
    at __docDbMain (getCustomerId.js:15:5)
    at Global code (getCustomerId.js:1:2)

So we’ve now tested out the old customer ID format, the new customer ID format, and some invalid input, and the UDF behaves as we expect.

Summary

UDFs provide us with a way to encapsulate simple computational logic, and to expose this within queries. Although we can’t refer to other documents or external data sources, UDFs are a good way to expose certain types of custom business logic. In this post, we’ve created a simple UDF for Cosmos DB, and tested it using a couple of simple documents. In the next part of this series we’ll move on to stored procedures, which allow for considerably more complexity in our server-side code.

Key Takeaways

  • UDFs are intended for simple computation.
  • They can be used within queries, including in the SELECT and WHERE clauses.
  • UDFs cannot access anything within the Cosmos DB collection, nor can they access any external resources.
  • They can accept one or more parameters, which must be provided when calling the UDF.
  • The name of the JavaScript function does not have to match the name of the UDF, but to keep your sanity, I highly recommend keeping them consistent.
  • UDFs can be invoked from within a query by using the udf. prefix, e.g. SELECT udf.getCustomerId(c) FROM c.
  • You can view the code for this post on GitHub.

 

Cosmos DB Server-Side Programming with TypeScript – Part 1: Introduction

Cosmos DB is a NoSQL database provided as part of Microsoft’s Azure platform. Designed for very high performance and scalability, Cosmos DB is rapidly becoming one of the default data storage options I recommend for new green-field applications and microservices. It is a fairly opinionated database, with some guidelines that you need to follow to take full advantage of its scalability and performance, but it also provides a number of features to enable sophisticated and powerful applications to be built on top of its engine.

One such feature is its server-side programmability model. Cosmos DB allows for stored procedures, triggers, and user-defined functions to run within its database engine. Interestingly, these are written using JavaScript and uploaded to the Cosmos DB collection in which they will run. Server-side programming gives a lot of extra power to a Cosmos DB-based application, including the ability to run transactions across multiple documents within the collection. In fact, server-side programming is the only way to get transaction semantics within Cosmos DB.

In this series of blog posts, we will explore server-side programming in Cosmos DB, and we will use TypeScript to write the server-side code. I will focus on how to build real-world applications, including adding unit tests to ensure the code behaves as expected, and incorporating the build and deployment of Cosmos DB server-side code into your CI/CD process. The series is split into six parts:

  • Part 1 (this post) gives an overview of the server side programmability model, the reasons why you might want to consider server-side code in Cosmos DB, and some key things to watch out for.
  • Part 2 deals with user-defined functions, the simplest type of server-side programming, which allow for adding simple computation to queries.
  • Part 3 talks about stored procedures. These provide a lot of powerful features for creating, modifying, deleting, and querying across documents – including in a transactional way.
  • Part 4 (coming soon) introduces triggers. Triggers come in two types – pre-triggers and post-triggers – and allow for behaviour like validating and modifying documents as they are inserted or updated, and creating secondary effects as a result of changes to documents in a collection.
  • Part 5 (coming soon) discusses unit testing your server-side scripts. Unit testing is a key part of building a production-grade application, and even though some of your code runs inside Cosmos DB, your business logic can still be tested.
  • Finally, part 6 (coming soon) explains how server-side scripts can be built and deployed into a Cosmos DB collection within an automated build and release pipeline, using Microsoft Visual Studio Team Services (VSTS).

In this series I presume some basic knowledge of Cosmos DB. If you’re completely new to Cosmos DB then I recommend reading Microsoft’s overview, and following along with one of the quick starts. A passing familiarity with TypeScript will also be helpful, but even if you don’t know how to use TypeScript, I’ll try to cover the key points you need to know to get started.

Using TypeScript

TypeScript is a language that compiles (or, technically, transpiles) into JavaScript. It provides a number of nice features and improvements over JavaScript, the main one being type safety. This means that the TypeScript compiler can check that your code is accessing the correct types and members as you write it. Writing code in TypeScript allows for a better level of certainty that your code is actually going to work, as well as providing some very nice development-time features such as IntelliSense. It also helps to have strong typing when unit testing, and particularly when mocking out external interfaces and classes within tests.

Because TypeScript compiles into JavaScript code, any JavaScript runtime will be able to run code that had been written in TypeScript. This includes Cosmos DB’s JavaScript engine, Chakra. Even though Cosmos DB doesn’t know about TypeScript or support it directly, we can still take advantage of many of the features that TypeScript provides, and then compile our script into JavaScript before handing it over to Cosmos DB for execution.

TypeScript also lets us separate out our code into multiple .ts files, keeping it tidy and well-organised. Cosmos DB requires that our code be in a single .js file, though – but thankfully, TypeScript can be configured to combine our code when it compiles it.

When working with external libraries and APIs within TypeScript, we need to use type definitions. These specify the details of the types we will use. While the Cosmos DB team doesn’t provide first-party type definitions for their server-side API, there are publicly accessible, open-source type definitions available from the DefinitelyTyped repository. We will use these later in this series.

Note that Cosmos DB supports the ECMAScript 2015 version of JavaScript. TypeScript can be configured to emit JavaScript in several different versions, including ECMAScript 2015 code, so this is not a problem for us. We’ll see how to do this in part 2 of this series.

Impact of Server-Side Programming on Request Units

When using Cosmos DB, we don’t provision CPU cores or disk speed or memory. Instead, Cosmos DB uses request units as its currency. A Cosmos DB collection is provisioned with a certain number of request units per second (RU/s), which can be scaled as necessary to cope with your application’s demands. The RU/s provisioned dictates the monetary cost of running the collection. For example, a simple collection with a light query load might be provisioned with 1000 RU/s, which (as of January 2018) costs approximately USD$60 per month. For more information on Cosmos DB’s request unit model see here, and for the latest pricing information, see here.

Server-side code running within Cosmos DB can easily consume a lot of request units, potentially exhausting your allowance for that second and forcing your application to have to retry operations against Cosmos DB. Furthermore, the cost of running queries server-side may sometimes be higher than the cost of running the equivalent query using the standard client-side APIs, due to the resources it takes to start up a stored procedure or function from JavaScript. Cosmos DB does have some optimisations to reduce the cost of running JavaScript code – for example, internally Cosmos DB compiles the JavaScript code to bytecode and then caches this bytecode so that it doesn’t need to recompile it on every invocation. However, running arbitrary code will usually be more expensive than just using the client-side query APIs, and this means that server-side code may not be appropriate if you don’t actually need the benefits it provides.

Additionally, the request unit usage for a given piece of server-side code is not fully predictable or consistent – in my own testing, I’ve seen the exact same piece of code, working on the same data set, take anywhere from 3.2 RUs through to 4.8 RUs to execute. This is in contrast to the rest of Cosmos DB, where request unit usage is very predictable.

Nevertheless for some scenarios, such as bulk inserts of multiple documents, or generating sample data, it may take fewer request units to run code server-side than client side. It is important to benchmark your code and compare the possible approaches to fully understand the best option for your requirements.

Consistency: Transactions and Indexes

Cosmos DB’s client-side programming model does not provide for transactional consistency across multiple documents. For example, you may have two documents to insert or update, and require that either both operations succeed or – if there is a problem with writing one of them – that both of them should fail. The Cosmos DB server-side programmability model allows for this behaviour to be implemented, because all server-side code runs within an implicit transaction. This means that we get ACID transaction semantics automatically whenever we execute a stored procedure or trigger. Cosmos DB runs stored procedures and triggers on the primary replica that is used to host the data, which allows it to give this level of transactional isolation while still allowing for high performance operations within the transaction.

Note, however, that transactions are not serialised. This means that other transactions may be happening simultaneously on other documents within the collection in parallel with your transaction. This is important because it means that functionality like real-time aggregation of data may not always be looking at a consistent view of the world, and you can get race conditions. This is simply due to the way Cosmos DB works, and is not something that we can easily program around.

A further nuance to be aware of is that, in Cosmos DB, indexes are updated asynchronously. This means that if you query the collection within a trigger, you may not see the document currently being inserted or updated. Again, this makes it challenging to do certain types of queries (such as aggregations), but is a byproduct of Cosmos DB’s emphasis on enabling high performance and throughput.

API Models

Cosmos DB provides several API models to access data in your databases: SQL to use a SQL-based syntax; MongoDB for using the MongoDB client libraries and tools; Table to use the Azure Storage table API; and Gremlin to use the Gremlin graph protocol. All of these ultimately store data in the same way though, and all of them allow for Cosmos DB’s server-side programmability model.

In this series I will focus purely on the SQL API, but most of the same concepts can be applied to the other API models.

Restrictions

Cosmos DB has placed some restrictions on the types of operations that can be run from within the server. This is mostly to optimise the performance and security of the service.

Time restriction: each server-side operation has a fixed amount of time that it must execute within. The exact amount of time is not documented, but the server-side API provides some features to indicate when your script is approaching its limit. We will discuss this more in later parts of the series. It is important to build in this restriction when designing your stored procedures and triggers, and to avoid writing server-side code that will make high volumes of queries. Instead, if you batch these up across multiple stored procedure calls, you are more likely to have all of your code execute successfully.

Limited set of functionality: although it executes arbitrary JavaScript, Cosmos DB’s server-side programming model is designed for basic computation and for interacting with the Cosmos DB collection itself. We cannot call external web APIs, communicate with other collections, or import any complex JavaScript libraries.

Limited fluent query capability: Cosmos DB’s server-side API has a fluent JavaScript-based query syntax to perform various types of queries, filters, projections, etc on the underlying collection. However, this API does not support all of the rich functionality that the SQL grammar provides, nor does it allow for the same types of queries as the other API models, such as Gremlin’s graph query capability.

One example feature that is missing from the server-side query API is aggregation. The Cosmos DB SQL dialect allows for queries such as SUM, MIN, MAX, and COUNT. These cannot be performed using the fluent server-side query API. However, SQL queries can be executed from within server-side code, so this is not a serious limitation and really just affects the way the code is written, not the functionality that is exposed.

Single JavaScript file: a single stored procedure, trigger, or user-defined function is represented by a JavaScript function, which in turn may call other functions. However, all of the code must be placed in a single file. JavaScript modules and other similar features are not supported. We will see how to split our functionality into multiple TypeScript files, while still emitting a single JavaScript file, later in this series.

Following Along

In this series, you will be able to follow along and create each type of server-side programming entity in Cosmos DB: a user-defined function, a stored procedure, and a trigger. We will build up a set of server-side code, and then in parts 5 and 6 of this series we will look at how to get these ready for a production deployment by testing and automatically building and deploying them to Cosmos DB.

You can follow along whether you use Windows, macOS, or Linux to develop. There are just a few prerequisites:

  • A good text editor: I use Visual Studio Code, which comes with the TypeScript programming extension, but you can use anything you like.
  • Node Package Manager (NPM): you can install this here if you don’t already have it.
  • An Azure subscription. Alternatively, you can use the Cosmos DB emulator to run this locally and at no charge, but you will need to adapt the instructions slightly.

Sample Scenario

A common use for a database is to store information about orders that customers make for products. Orders typically contain some basic overall information, such as an order ID, date, customer ID, and a set of order items – references to products and the quantities ordered. In this series we will work with a simple hypothetical order database implemented in Cosmos DB.

We will use the SQL API, and we will use a non-partitioned collection. Note that partitioned collections behave the same way as non-partitioned collections when it comes to server-side programmability, but they also have a few nuances in their behaviour that we won’t go through here.

Summary

Server-side programming in Cosmos DB is extremely powerful. It gives us the ability to write functions, stored procedures, and triggers that execute within the database engine, and allow for features that are simply not possible through the client-side programming model. However, there are limitations and things to be aware of, including the potentially high cost of running some types of operations from the server. The features also do not provide the same degree of flexibility and power as their counterparts in SQL Server and other relational databases. Nevertheless, the server-side programming model in Cosmos DB is enormously useful for certain types of situations.

By using TypeScript to add type safety, and by adding unit tests and good continuous integration and continuous deployment practices, we can build advanced behaviour into our production-grade applications – all while taking advantage of the high performance and scale capabilities of Cosmos DB.

In the next part of this blog series, we will start writing some server-side code – first by building a user-defined function.

Performing OCR with Azure Cognitive Services and HTML5 Media Capture API

There are a few ways to access camera on mobile devices during application development. In our previous post, we used the getUserMedia API for camera access. Unfortunately, as of this writing, not all browsers support this API, so we should provide a fallback approach. On the other hand, HTML5 Media Capture API is backed by almost of all modern browsers, which we can utilise with ease. In this post, we’re going to use Vue.js, TypeScript and ASP.NET Core to build an SPA that performs OCR using Azure Cognitive Service and HTML Media Capture API.

The sample code used in this post can be found here.

Vision API – Azure Cognitive Services

Cognitive Services is an intelligence service provided by Azure and uses machine learning resources. Vision API is a part of Cognitive Services to analyse pictures and videos. It performs analysis of expressions, ages and so on from someone or something in pictures or videos. It even extracts texts from them, which is OCR feature. Previously it was known as Project Oxford, while it was renamed to Cognitive Services when it came out for public preview. Therefore, the NuGet package still has its title of ProjectOxford.

HTML Media Capture API

Media Capture API is one of HTML5 features. It enables us to access to camera or microphone on our mobile devices. According to the W3 document, this is just an extension of the existing input tag with the type="file" attribute. Hence, by adding both accept="image/*" and capture="camera" attributes to the input tag, we can use the Media Capture API straight away on our mobile devices.

Of course, this doesn’t interrupt on existing user experiences for desktop browsers. In fact, this link confirms how the Media Capture API works on either desktop or mobile browsers.

ASP.NET Core Web API

The image file passed from the front-end side is handled by the IFormFile interface in ASP.NET Core.

Well, theory is enough. Let’s make it!

Prerequisites

  • ASP.NET Core application from the previous post
  • Computer, tablet or smart phone having camera

Implementing Vue Component – Ocr.vue

First of all, we need a Vue component for OCR. This component is as simple as to have an input element, a button element, an img element, and a textarea element.

If we put the ref attribute on each element above, the Vue component can directly handle it. The button element binds the onclick event with the event handler, getText. Ocr.ts contains the actual logic to pass image data to the back-end server.

Like this previous post, in order to use dependency injection (DI), we create a Symbols instance and use it. axios is injected from the top-most component, App.vue, which will be touched later in this post.

We also create a FormData instance to pass the image file extracted from the input element, through an AJAX request. This image data then will be analysed by Azure Cognitive Services.

Updating Vue Component – Hello.vue

Ocr.vue is now combined with Hello.vue as a child component.

Dependency Injection – App.vue

The axios instance is provided at the top-most component, App.vue, which is consumed its child components. Let’s see how it’s implemented.

We use the symbol instance as a key and provide it as a dependency.

Everything on the front-end side is done. Let’s move onto the back-end side.

Subscribing Azure Cognitive Service

We need to firstly subscribe Azure Cognitive Service. This can be done through Azure Portal like:

At the time of this writing, Azure Cognitive Services are in public preview, which we only can choose the West US region. Choose Computer Vision API (preview) for API Type and F0 (free) for Pricing Tier. Make sure that we only can have ONE F0 tier in ONE subscription for ONE API type.

It takes about 10 minutes to activate the subscription key. In the meantime, let’s develop the actual logic.

Developing Web API – ProjectOxford Vision API

This is relatively easy. Just use the HttpClient class to directly call REST API. Alternatively, ProjectOxford – Vision API NuGet package even makes our lives easier to call the Vision API. Here’s the sample code.

The IFormFile instance takes the data passed from the front-end through the FormData instance. For some reason, if the IFormFile instance is null, the same data sitting in the Request.Form.Files also needs to be checked. Put the API key to access to the Vision API. The VisionServiceClient actually returns the image analysis result, which is included to the JSON response.

We’ve completed development on both front-end and back-end sides. Let’s run this app and access it from our mobile device. The following video clip shows how iPhone takes a photo, sends it to the app, and gets the result.

So far, we’ve briefly looked at Azure Cognitive Services – Vision API for OCR implementation. In fact, depending on the original source images, the analysis quality varies. In the video clip above, the result is very accurate. However, if there are outlines around the text, or contrast between text and its background is very low, the quality significantly drops. In addition to this, CAPTCHA-like images don’t return satisfactory results. Once after Cognitive Services performs enough learning with substantial number of sources, the quality becomes high. It’ll be just matter of time.

Dependency Injection in Vue.js App with TypeScript

Dependency management is one of critical points while developing applications. In the back-end world, there are many IoC container libraries that we can make use of, like Autofac, Ninject, etc. Similarly, many modern front-end frameworks also provide DI features. However, those features work way differently from how back-end libraries do. In this post, we’re going to use TypeScript and Vue.js for development and apply an IoC container library called InversifyJS that offers very similar development experiences to back-end application development.

The code samples used in this post can be found here.

provide/inject Pair in VueJs

According to the official document, vue@2.2.0 supports DI feature using the provide/inject pair. Here’s how DI works in VueJs. First of all, declare dependency, MyDependency in the parent component like:

Then its child component consumes the dependency like:

Maybe someone from the back-end development got a question. Child components only consumes dependencies that are declared from their parent component. In other words, in order for all components to consume all dependencies, this declaration MUST be done at the top-level component of its hierarchy. That’s the main difference between VueJs and other back-end IoC containers. There’s another question – VueJs doesn’t provide a solution for inter-dependency issue. This inter-dependency should be solved by a third-party library. But that’s fine. We’re going to use TypeScript anyway, which has a solution for the inter-dependency issue.

DI in VueJs and TypeScript

Evan You, the creator of VueJs, has recently left a comment about his design philosophy on VueJs framework.

While using a class-based API by default may make it more “friendly” to devs used to classes, it also makes it more hostile to a large group of users who use Vue without build tools or transpilers. When you are advocating your preference, you might be missing some nuance we have to take into account as a framework.

This is why we offer the object-based API as the baseline and the class-based API as an opt-in. This allows us to cater to both groups of users.

Therefore, we need to sort out either using the provide/inject pair or using another approach, ie. service locator pattern. In order to use the provide/inject pair, as we found above, we need to put an IoC container instance at the top-level of component. On the other hand, we can simply use the container as a service locator. Before applying either approach, let’s implement the IoC container.

Building IoC Container using InversifyJS

InversifyJS is a TypeScript library for IoC container, which is heavily influenced from Ninject. Therefore syntax is very similar to each other. Interface and class samples used here is merely modified from both libraries’ conventions – yeah, the ninja stuff!

Defining Interfaces

Let’s define Weapon and Warrior interfaces like below:

Defining Models

InversifyJS uses Symbol to resolve instances. This is a sample code to define multiple symbols in one object. This object contains multiple symbols for Warrior, Weapon and Container.

The @injectable decorator provided by InversifyJS defines classes that are bound into an IoC container.

The @inject decorator goes to constructor parameters. Make sure that those parameters require the Symbol objects defined earlier.

Make sure that we should use the same Symbol object defined earlier. If we simply use Symbol("Weapon") here, it wouldn’t be working as each Symbol object is immutable.

Implementing IoC Container

Let’s implement the IoC container using the interfaces and models above.

The last part of the code snippet above, container.bind(...).to(...), is very similar to how IoC container works in C#. Now we’re ready for use of this container.

Attaching Child Component

Unlike the Previous Posts, We’re adding a new child Vue component, Ninja.vue to Hello.vue for dependency injection.

Hello.vue has got the Ninja.vue component as its child. Let’s have a look at the Ninja.vue component.

Now, let’s apply both service locator and provide/inject pair.

Applying Service Locator

We’re updating the Ninja.vue to use service locator:

As we can see above, the IoC container instance, container is directly consumed within the Ninja.vue component. When we run the application, the result might be looking like:

As some of us might uncomfortable to use the service locator pattern, now we’re applying the built-in provide/inject pair.

Applying provide/inject Pair

As we identified above, in order to consume all dependencies at all Vue components, we should declare IoC container as a dependency at the top-level of the component, ie) App.vue.

We can see that the container instance is provided with the symbol, SERVICE_IDENTIFIER.CONTAINER defined earlier. Now let’s modify the Ninja.vue component:

The @Inject decorator takes care of injecting the container instance from the App.vue component. Make sure that the same symbol, SERVICE_IDENTIFIER.CONTAINER is used. All good! Now we can see the same result like the picture above.

So far, we’ve had an overview how to use DI in VueJs & TypeScript app in two different approaches – service locator or provide/inject pair. Which one to choose? It’s all up to you.

Accessing to Geolocation on Mobile Devices from ASP.NET Core Application in Vue.js and TypeScript

In the previous post, we used HTML5 getUserMedia() API to access camera on our mobile devices. In this post, we’re using geolocation data on our mobile devices.

The code samples used for this post can be found here.

navigator.geolocation API

Unlike getUserMedia() API, geolocation API has a great level of compatibility of almost all browsers.

Therefore, with a simple TypeScript code, we can easily use the geolocation data.

NOTE: In order to use the geolocation API, the device must be connected to the Internet. Also, each browser vendor uses its own mechanism to get geolocation data, which will cause different result even in the same device. This article gives us more details.

Prerequisites

  • ASP.NET Core App from the previous post
  • Computer or mobile devices that can access to the Internet through Wi-Fi or mobile network

NOTE 1: We use vue@2.2.2 and typescript@2.2.1 in this post. There are breaking changes on VueJs for TypeScript, so it’s always a good idea to check out the official guideline.

NOTE 2: Code samples used in this post were from the MDN document that was altered to fit in TypeScript.

Updating Hello.vue

In order to display latitude, longitude and altitude retrieved from the geolocation API, we need to update the Hello.vue file:

That’s pretty much self descriptive – clicking or tapping the Get Location button will display those geolocation data. Let’s move onto the logic side.

Updating Hello.ts

The Get Location button is bound with the getLocation() event, which needs to be implemented like below:

First of all, we need to declare properties for latitude, longitude and altitude, followed by the getLocation() method. Let’s dig into it.

  • First of all, we check the navigator.geolocation isntance if the web browser supports geolocation API or not.
  • Call getCurrentPosition() method to get the current position. This method then passes two callback methods and an option instance as its parameters.
  • Callback method, success(), passes the position instance containing current position details and binds co-ordinates to the browser.
  • error() callback handles error.
  • options instance provides options for the geolocation API.

NOTE Each callback method has its return type, according to the type definition, which is not necessary. Therefore, we just return null

The options instance used above is actually an interface type of PositonOptions that needs to be implemented. Its implementation might be looking like below:

We completed the TypeScript part. Let’s run the app!

Results

When we use a web browser within our dev machine, it firstly asks us to get a permission to use our location data:

Click Allow and we’ll see the result.

This time, let’s do it on a mobile browser. This is taken from Chrome for iPhone. It also asks us a permission to use geolocation data.

Once tapping the OK button, we can see the result.

So far, we’ve briefly looked at the geolocation API to populate current location. That’s not that hard, isn’t it?

If we have more complex scenario, need more accurate location details, or need constant access to the location data even we’re not using the app, then native app might have to be considered. Here’s a good discussion regarding to these concerns. But using HTML5 geolocation API would be enough in majority of cases.

Accessing to Camera on Mobile Devices from ASP.NET Core Application in Vue.js and TypeScript

In the previous post, we built an ASP.NET Core application using Vue.js and TypeScript. As a working example, we’re building a mobile web application. Many modern web browsers supporting HTML5 can access to multimedia devices on users’ computer, smartphones or tablets, such as camera and microphone. The Navigator.getUserMedia() API enables us to access to those resources. In this post, we’re actually going to implement a feature for camera access on our computer and mobile devices, by writing codes in VueJs and TypeScript.

The code samples used for this post can be found here.

getUserMedia() API

Most modern web browsers support this getUserMedia() API, as long as they support HTML5. There are two different APIs around this method – one is Navigator.getUserMedia() that supports callback functions, while the other MediaDevices.getUserMedia(), that came up later, supports Promise so that we can avoid Callback Hell. However, not all browsers support the MediaDevices.getUserMedia(), so we need to support both anyway. For more details around getUserMedia(), we can find some practical samples in this MDN document.

Prerequisites

  • ASP.NET Core application from the previous post
  • Computer, tablet or smartphone having camera

NOTE 1: This post uses VueJs 2.2.1 and TypeScript 2.2.1. VueJs 2.2.1 introduced some breaking changes how it interacts with TypeScript. Please have a look at the official guide document.

NOTE 2: vue-webcam written by @smronju was referenced for camera access, and modified to fit in the TypeScript format.

Update Hello.vue

We need a placeholder for camera access and video streaming. Add the following HTML codes into the template section in Hello.vue.

  • video accepts the camera input. src, width, height and autoplay are bound with the component in Hello.ts. Additionally, we add the ref attribute for the component to recognise the video tag.
  • img is where the camera input is rendered. The photo field is used for data binding.
  • button raises the mouse click or finger tab event by invoking the takePhoto function.

The HTML bits are done. Let’s move on for TypeScript part.

Update Hello.ts

The existing Hello.ts was simple, while this time it’s grown up to handle the camera API. Here’s the bits:

We can see many extra data fields for two-way data binding between user input and application. Some of them comes with their default values so that we don’t have to worry about their initialisation too much.

  • The takePhoto() function creates a virtual DOM for canvas, converts the input signal from the video into an image, and sends it to the img tag to display snapshot.

  • The mounted() event function is invoked when this component, Hello.ts, is mounted to its parent. It uses the getUserMedia() API to bind streaming source to the video tag.
  • The video tag through this.$refs.video is the HTML element that has the ref attribute in Hello.vue. Without the ref attribute, VueJs cannot know where to access to the tag.

NOTE: The original type of the this.$refs instance is { [key: string]: Vue | Element | Vue[] | Element[] }, while we cast it to any. This is to avoid build failure due to the linting error caused by using the original type and accessing to the resource by referencing like this.$refs.video. If we don’t want to cast it to any, we can use this.$refs["video"] instead.

We’ve so far completed the coding part. Now, let’s build this up and run a local IIS Express, and access to the web app through http://localhost:port. It works fine.

This time, instead of localhost, use the IP address. If we want to remotely access to our local dev website, this post would help.

It says we can’t use the camera because of its insecure access. In order to use the getUserMedia() API, we should use HTTPS connection to prevent private data exposure. This only happens when we’re using Google Chrome, not FireFox or Edge. So, just change the connection to HTTPS.

Now we can use IP address for camera access. Once we allow it we can immediately see our face directly on the web like below (yeah, it’s me! lol).

Let’s try this from our mobile devices. The first one is taken from Android phone, followed by the one taken from Windows Phone, then the ones from iPhone. Thanks Boris for help take those pictures!

Errr… what happened on iPhone? The camera is not accessible from both Safari for iOS and Chrome for iOS!!

This is because not all mobile web browsers support the getUserMedia() API.

getUserMedia Browser Compatibility

Here’s the data sheet from http://mobilehtml5.org/.

Unfortunately, we can’t use the getUserMedia API on iOS for now. For iOS users, we have to provide alternative methods for their user experience. There’s another API called HTML Media Capture that is supported by all mobile web browsers. It uses the traditional input type="file" tag. With this, we can access to camera on our mobile devices.

In the next post, we’re going to figure out how to provide a fallback option, if getUserMedia() API is not available.

Writing Vue.js Applications in TypeScript on ASP.NET Core

In the previous post, we’ve briefly walked through how to build Vue.js application on ASP.NET Core. Like other modern JavaScript framework, VueJs also supports TypeScript out-of-the-box. If we can get full benefits from TypeScript to build a VueJs app, it would be awesome! There are many resources referring to the combination of VueJs and TypeScript. However, they are not using the basic template that VueJs provides, which brings about less confidence to those developers who just started using VueJs. Even worse, due to the recent version up of Webpack to 2.x, we might need a new tutorial to build a VueJs application using TypeScript. In this post, our goal will be:

  • To use the basic template provided by VueJs,
  • To use Webpack version 2.x, and
  • To run the app on ASP.NET Core.

The sample code used in this post can be found at here.

Prerequisites

We have already built a VueJs application running on ASP.NET Core in the previous post. So we’re going to re-use that.

Update on March 6th, 2017: We updated the TypeScript version to 2.2.1 for this post.

Installing npm Packages

TypeScript

We can install TypeScript locally only for this application:

Or we can install it globally:

If TypeScript is installed globally, we should link it to this application:

ts-loader

ts-loader offers us to load .ts files to .js file without actually building them during the development time.

vue-class-component & vue-property-decorator

If we want to use .ts in our VueJs development, as the official document recommends, we should install the vue-class-component library for class decorators.

It may be necessary to install vue-property-decorator to extend vue-class-component. This is not relevant to this post, though.

vue-typescript-import-dts

TypeScript needs type definitions. vue-typescript-import-dts helps recognise .vue files as .ts.

All necessary npm packages are installed. Let’s move on.

Configurations for TypeScript

tsconfig.json

In order to use .ts, we firstly need tsconfig.json. In this post we just use the bare minimum settings to work. Further details about tsconfig.json can be found here.

Let me explain the configuration in-depth.

  • VueJs supports ECMAScript 5. Therefore, we need to target TypeScript to es5. It means that module should be CommonJs as well as lib should include dom, es2015 and es2015.promise.
  • types declares custom type definitions. As we’ve installed vue-typescript-import-dts, include it here so that the application can recognise .vue files as .ts files.
  • In order to use class decorators, we’ve installed vue-class-component. But this is not enough. We need to enable it by setting the experimentalDecorators value to be true.
  • Within the include property, we need to declare which directories are considered containing .ts files.

Update on March 6th, 2017 Due to the version update of VueJs to 2.2.x, tsconfig.json also needs to be updated. This is the recommended configuration from the official guide.

Also, please make sure that we create the template from vue-cli by running vue init webpack. It installs vue@2.2.1 and vue-router@2.2.0. If those versions are different, please update them.


.eslintignore

While developing apps in TypeScript, .js files are automatically compiled and generated. But it’s not guaranteed those files comply to linting process. Therefore, we can’t be sure if those generated .js files are ESLint compliant or not. Therefore, to avoid linting errors from those generated .js files, we just turn it off by adding a line, src/**/*.js, to .eslintignore.

We just completed basic configurations for TypeScript compiling. Let’s move on.

Converting JavaScript to TypeScript

It’s time to convert existing .js files in the src directory to .ts ones situated. We’ll only look after both build and src directories.

build/webpack.base.conf.js

As Webpack is the only service to refer this file, so it’s not necessary to change this to .ts. But we do need to modify it.

First of all, the entry point should be changed from main.js to main.ts:

Then we need to replace the babel-loader part with the ts-loader one:

Every .ts file is handled by this loader. Here’s an interesting option, appendTsSuffixTo. If we use this, .vue files can be treated as .ts ones. VueJs uses the Single File Component approach – all HTML section, JavaScript section, and CSS sections are put in one single file called .vue. Therefore we need to handle it to be a TypeScript file, particularly for the JavaScript section.

We’ve completed webpack configuration to enable TypeScript handling. Let’s really convert JavaScript files to TypeScript ones.

src/main.jssrc/main.ts

Change the existing JavaScript syntax to the one for TypeScript like:

Spot on the new Vue({ ... }) part. Instead of template and components, the render function is placed. Everything has been compiled before hitting this point, and each component needs more control by itself, so we just use the render function. For more details about the render function, please refer to the official document.


Updated on March 6th, 2017: Due to the version update of VueJs to 2.2.x, the import statements part needs to be updated like below:


src/router/index.jssrc/router/index.ts

We don’t have to worry about this. Just be cautious when using import ....


Updated on March 6th, 2017: Due to the version update of VueJs to 2.2.x, the import statements part needs to be updated like below:


src/App.vuesrc/App.ts

Instead of using one single .vue file, we’re separating the TypeScript part from each .vue. Why are we doing this, by the way? We can still use .vue indeed. But for better maintainability, we’d better to create a separate .ts file. Let’s have a look how we can implement App.ts that is extracted from App.vue.

@Component decorator contains the name declaration so that the router can easily recognise it. The script part in the original App.vue can be altered like this:

Make sure that we should include lang="ts" as an attribute of the script tag.


Updated on March 6th, 2017: Due to the version update of VueJs to 2.2.x, the import statements part needs to be updated like below:


src/components/Hello.vuesrc/components/Hello.ts

Now, we’re going to extract the script section from Hello.vue to Hello.ts. Let’s have a look.

Likewise, @Component contains the name declaration. Previously all two-way binding fields were defined within the data function. Using properties makes them more class-friendly. Functions became methods.

Maybe someone indicates a small change, comparing to the previous post. In order to use AJAX requests and responses, we used vue-resource. However, it’s changed to axios. According to the official VueJs blog post, vue-resource is no more supported as an official VueJs extension. Instead axios is recommended because of its richer features. In addition to this, axios provides TypeScript definitions, so there’s no reason not to use this. Its usage is almost identical to vue-resource.

Once Hello.ts is extracted, the original Hello.vue should now be changed to:


Updated on March 6th, 2017: Due to the version update of VueJs to 2.2.x, the import statements part needs to be updated like below:


All done for conversion from JavaScript to TypeScript! It seems that we’ve done fairly massive conversion. The basic template is optimised to JavaScript, so what we’ve done so far is basically the conversion job. From now on, we can write all logic using TypeScript!

Press F5 key on your Visual Studio to run the application and see the result.

The right-hand side of the window on the picture above is Vue.js devtools, which is a Chrome extension. When we install it, we can use it right away through Chrome’s Developer Tools.

One More Thing …

So far, we’ve done the conversion of VueJs to TypeScript. As this is for local development environment, we need one last modification for deployment. Here’s the overall process of building applications for deployment:

  1. To compile .ts files and generate corresponding .js ones.
  2. To modularise and build bundles through webpack.
  3. To build ASP.NET Core libraries.
  4. To generate an artifact for deployment to Azure or IIS.
  5. To deploy.

By updating package.json and project.json we can easily achieve this goal.

package.json

Within package.json, the scripts was originally looking like:

We need to add another one for TypeScript compilation. Let’s change it like:

  • build:ts is to compile .ts files.
  • build:main is to be responsible for existing build.
  • build is to change to call both build:ts and build:main consecutively.
  • --no-deprecation flag may bring an attention. When compiling, ts-loader throws a deprecation warning. It’s OK but Visual Studio treats it as an error so build/deploy fails. By providing this flag will enable build/deploy through Visual Studio successfully.

project.json

Finally, open project.json to confirm the prepublish section.

All good now! After the deployment to Azure Web App, we can see the following screen:

Of course, if CI/CD is preferred, we can simply use dotnet publish feature.

We’ve so far had a quick look to write a VueJs application in TypeScript, bundle it on ASP.NET Core and deploy it to Azure. As mentioned earlier, the very first part is a bit complicating but it’s not that different from normal TypeScript development. Let’s build a real world application using VueJs and TypeScript!!

Building Applications with Event Sourcing and CQRS Pattern

When we start building an application on cloud, like Azure, we should consider many factors. Those factors include flexibility, scalability, performance and so on. In order to satisfy those factors, components making up the application should be loosely coupled and ready for extension and change at any time. For those considerations, Microsoft has introduced 24 cloud design patterns. Even though they are called as “Cloud Design Patterns”, they can be used just for application development anyway. In this post, I’m going to introduce Event Sourcing Pattern and CQRS Pattern and how they can be used in a single page application (SPA) like AngularJS application.

The complete code sample can be found here.

Patterns Overview

I’m not going into too much details here to explain what Event Sourcing (ES) Pattern and CQRS Pattern are. According to articles linked above, both ES and CQRS easily get along with each other. As the name itself says, CQRS separates commands from query – commands and query use different dataset and ES supports event stream for data store (commands), and materialisation and replaying (query). Let’s take a look at the diagram below.


[Image from: https://msdn.microsoft.com/en-us/library/dn589792.aspx]

This explains how ES and CQRS work together. Any individual input (or behaviour) from a user on the presentation layer (possibly Angular app in this post) is captured as an event and stored into event stream with timestamp. This storing action is append-only, ie events are only to be added. Therefore, the event stream becomes a source of truth, so all events captured and stored into the event stream can be replayed for query or materialised for transaction.

OK. Theory is enough. Let’s build an Angular app with Web API.

Client-side Implementation for Event Triggering

There are three user input fields – Title, Name and Email – and the Submit button. Each field and button acts as an event. In this code sample, they are named as SalutationChangedEvent, UsernameChangedEvent, EmailChangedEvent and UserCreatedEvent. Those events are handled by event handlers at the Web API side. What the Angular app does is to capture the input values when they are being changed and clicked. This is a sample TypeScript code bits for the name field directive.

This HTML is a template used for the directive below. ng-model will capture the field value and the value will be sent to the server to store event stream.

Please bear in mind that, as this is written in TypeScript, the coding style is slightly different from the original Angular 1.x way.

  1. The interface IUserNameScope defines model property and change function. This inherits $scope.
  2. The interface is injected to both link and controller of the directive UserName that implements ng.IDirective.
  3. A link function of the directive takes care of all DOM related ones.
  4. The link function calls the function declared in $scope to send AJAX request to Web API.
  5. A POST AJAX request is sent through userNameFactory to the server.
  6. A response comes from the server as a promise format and the response is passed to replayViewFactory for replay.

Both Title and Email fields work the same way as the Name field. Now, let’s have a look how the replay view section looks like.

This HTML template is used for the directive below. The following directive is only to replay responses.

As you can see, this directive only calls the replayViewFactory.getReplayedView() function to display what changes are. How do those events get consumed at the server-side then? Let’s move onto the next look.

Server-side Implementation for Event Processing

The POST request has been sent through a designated endpoint like:

This request is captured in this Web API action:

The action in the controller merely calls the this._service.ChangeUsernameAsync(request) method. Not too excited. Let’s dig into the service layer then.

  1. Based on the type of the request passed, an appropriate request handler is selected.
  2. The request handler converts the request into a corresponding event. In this code sample, the UsernameChangeRequest is converted to UsernameChangedEvent by the handler.
  3. An event processor takes the event and process it.

A question may arise here. How does request handler selection work? Each request handler implements IRequestHandler and it defines two methods:

Therefore, you can create as many request handlers as you like, and register them into your IoC container (using Autofac for example) like:

In the sample code used here registers five request handlers. If your business logic is way far complex and require many request handlers, you might need to consider moduling those request handlers automatic registration. I’ll discuss this in another post soon. Another question may arise again. How does the event processor work? Let’s have a look. Here’s the event processor:

This is quite similar to the EventStreamService.ChangeUsernameAsync(). First of all, find all event handlers that can handle the event. Then those selected event handlers process the event as all event handlers implements IEventHandler interface:

To wrap up,

  1. A user action is captured at a client-side and passed to a server-side as a request.
  2. The user action request is converted to an event by request handlers.
  3. The event is then processed and stored into event stream by event handlers.

Of course, I’m not arguing this is the perfect example for event processing. However, at least, it’s working and open for extension, which is good.

Replaying Events

Now, all events are raised and stored into event stream with timestamp. Event stream becomes a source of truth. Therefore, if we want to populate a user’s data against a particular time period, as long as we provide timestamp, we’re able to load the data without impacting on the actual data store. If you run the code sample on your local and make some user input change, you’ll actually be able to see the replayed view.

Now, let’s store the user data into the real data store by event materialisation.

Materialising Events

When you hit the Submit button, the server-side replays all events from the event stream with the current timestamp for materialisation. Then the materialised view is stored into the User table. As this is considered as another event, another event, UserCreatedEvent is created and processed by UserCreatedEventHandler. Unlike other event handlers, it does not only use the event stream repository, but also use the user repository.

In other words, the event itself is stored into the event stream and a user data from the event is stored into the user repository. Once stored, you will be able to find on the screen.

Please note that, if you change Title, Name, or Email but not yet click the Submit button, you’ll find some difference like the following screen:

So far, we’ve briefly discussed both ES pattern and CQRS pattern with a simple Angular – Web API app. How did you find it? Wouldn’t it be nice for your next application development? Make sure one thing. Applying those patterns might bring overly complex architecture into your application as there are many abstraction layers involved. If your application is relatively simple or small, you don’t have to consider those patterns. However, your application is growing and becomes heavier and complex, then it’s time to consider getting those patterns implemented for your application. Need help? We are Kloudie, the expert group.