Configuration – Part 2

Review

Last time we covered the construction of a .NET Core based REST web application that served up a shared collection of configuration values. The service also used a backchannel based on SignalR, for notifying clients about any changes to those shared configuration values. We wrote a custom provider for the Microsoft.Extensions.Configuration project, to tie everything together and hide the gory implementation details from our (.NET) clients. Finally, we covered the addition of an example client console application that demonstrated an end-to-end use of all the pieces.

What we didn’t do last time was run the example application and prove that all this code actually does something. Let’s do that now.

 

Running the Example

The first thing to do is open the solution in Visual Studio 2017 or higher. From there, we need to change the start up order of the applications so that the REST controller starts just prior to the example console application. To do that, right click on the solution in the Solution Explorer window, in Visual Studio, then choose the “Set StartUp Projects” menu choice.

SetStartupProjects

Doing that brings up a dialog that allows one to specify the project(s) to run, when the solution is started in the debugger.

StartOrder

Start by clicking the “Multiple startup projects” option, then arrange the order of your projects so that they match the order I’ve shown here. Make sure the action is set to “Start” for the host and example client projects. Click the OK button and we’re ready to run the software!

The next step is to hit F5 to run the project. You should see a console window and a browser window come up. They should look like this:

secondRun

Sit back for a moment and feel good about this. You’ve accomplished quite a bit! You have just performed an end-to-end demonstration of a .NET client using a shared configuration based on a custom REST controller and a custom configuration provider.

So, the next thing to do is demonstrate the notifications. The easiest way to do that is to use PostMan. At the time I wrote this, PostMan could be downloaded, for free, from [HERE]

After you install Postman, open it and configure it with the endpoint for the REST service. Choose the POST verb, then add a Content-Type header to the request with a value of application/json. Finally, add the JSON for the configuration key-value pair to the body. That should all look something like this:

postman

Pressing the send button send the POST request to the configuration controller, which responds by changing the value for the “SampleSection:Key1” setting in the collection of shared configuration settings, and also sends the notification through the SignalR connection.

We can prove that it works by looking at the example console window, which now looks like this:

changed

And that proves out the entire pipeline, end-to-end.

 

Outstanding Issues

There are two outstanding issues that we still need to deal with: (1) The provider we’re using to read our config.json file, on the REST service, won’t write our changes back to disk, and (2) the underlying Microsoft configuration API doesn’t have a facility for deleting configuration values.

The first issue is really only an issue if we want to continue storing our settings in a JSON file, which, probably isn’t the best choice for a production environment anyway. Either way though, the Microsoft provider doesn’t propagate writes back to the file, so, we’ll need to create another custom provider and add that functionality ourselves. The good news is, since we’re going to write a custom provider, we’ll get to decide what storage medium makes sense for us.

The second issue appears to be by-design and I imagine Microsoft left deletes off for a good reason. At least, I hope they left deletes off for a good reason. I could add the missing functionality by modifying the Microsoft NUGET package, but that would leave me with maintaining a custom branch and I don’t like that idea. I could also figure out a way to add the delete mechanism at the provider level, which would mean I could simply include it in our soon to be written custom provider and be done with it. The downside there is that deletes would stop working correctly if we ever switched to another provider. Personally, I can live with that, but, you may need to make that call for your own projects.

So, it looks like the next step is to create another custom provider for the Microsoft NUGET package. Since I figure most people will want to use a database as a storage medium, I’ll write the new provider to use ADO and store everything in a simple SQL-Server table. Let’s get started.

 

Second Provider

Just like we’ve done before, start by hitting the Ctrl+Shift+N key combination to bring up the “Add New” dialog.

AdoProject

Once that’s done, make sure the .NET Standard project type is chosen. Then, make sure the Solution dropdown contains “Add to solution”. Finally, name your new Class Library appropriately. Here I’ve named mine “CG.Configuration.Ado”. Press the OK button to add the new project to your solution.

The project contains a file named Class1.cs by default. You can remove that class, we won’t need it.

Our new project will need the “Microsoft.Extensions.Configuration” and “System.Data.Sqlclient” NUGET packages added, so go back into the NuGet Package Manager and add them now.

Much like we did last time, let’s start by adding the following new files to the project:

  1. A provider class, which we’ll put into a file named AdoConfigurationProvider.cs
  2. A source interface, which we’ll put into a file named IAdoConfigurationSource.cs
  3. A source class, which we’ll put into a file named AdoConfigurationSource.cs
  4. An extension class, which we’ll put into a file named ConfigurationBuilderExtensions.cs

Adding each file is as simple as using the Ctrl+Shit+A key combination (with the ADO provider project highlighted), and then choosing the right name and type, each time, in the Add New Item dialog. Afterwards, the project should look like this:

StartingAdoProj

Let’s start by modifying the IAdoConfigurationSource interface, like this:

We’ll need the configuration string property later, when we go to add this provider to our REST service. Next, let’s implement the corresponding AdoConfigurationSource class, like this:

Just like in our last provider, this class is mostly just an implementation of the IAdoConfigurationSource interface. Next we’ll implement the ConfigurationBuilderExtensions class, like this:

Just like last time, this extension method exists to clean up the code that injects the source into the builder, and remove it as a source of noise from the caller’s environment. This way, all the caller has to do is call a single method on the builder and we’ll handle the rest.

Let’s implement the provider class now:

The provider class is usually where all the action is and this one is no exception. Let’s go through the code one method at a time.

We’ll start with the constructor. It supplies the source reference from the Provider and stores that object in a property named “Source”. We’ll need that reference later to get at the “ConnectionString” setting in the other methods.

The next method is the Load method. This method is responsible for loading an internal data collection with configuration values. It starts by getting a reference to the connection string, through the Source property that we saved in the constructor. It then creates a SqlConnection object with that value. The connection is opened, then a SQLCommand object is created, passing in a SQL statement for selecting the values from our SQL-Server table. We use return the values as a reader. We loop and use that reader to populate KeyValuePair objects which are then added to the “Data” property, which houses the internal data collection.

The next method is the Set method. This method got somewhat complicated when I was writing it so I factored out some of the complexity into three other methods: DeleteKey, UpdateKey and InsertKey. I will go over those methods as well. The reason this method got so complicated is that we are doing two things at once here: (1) storing values into SQL-Server, and (2) implement our pseudo-delete feature. Keep that in mind as we look at this bit of code. The first thing the Set method does is get a reference to the connection string. It then creates a SqlConnection object with that value. The connection is opened, then we check to see if the key starts with the “~” symbol. That’s our special token that we use, internally, to denote a key that should be remove. If the key passed in as a parameter to the Set method contains that token, we delete it, otherwise, we either insert it or updated it. So, assuming the key contains the token, we call the DeleteKey method and then call ReloadToken to fire off any listening change tokens. On the other hand, if the hey doesn’t contain the token the we assume it will be an update and we call the UpdateKey method. That method, UpdateKey, returns a count of the number of rows that were updated in Sql-Server. That means, if the row doesn’t already exist the update count will be zero. If that return value is zero we assume the key doesn’t already exist in the Sql-Server table and we call InsertKey, to insert it.

The next method is UpdateKey. It starts by creating a SqlCommand using a SQL statement to update our SQL-Server table. The parameters are passed in and the command is executed, returning the update count from SQL-Server in the process.

The next method is the InsertKey method. It starts by creating a SqlCommand using a SQL statement to insert a key-value pair to our SQL-Server table. The parameters are passed in and the command is executed.

The last method is the DeleteKey method. It started by trimming the token from the front of the key. That has to happen since the key that was passed as a parameter still contains the token while all the keys in SQL-Server do not. After the key is trimmed, the method creates a SqlCommand using a SQL statement to delete a range of key value pairs from the SQL-Server table that roughly match the given key. That fuzzy match is important because the Microsoft configuration library is very JSON centric and it passes in sections separately from any values in that section. For instance, our sample config.json file had this JSON:

That JSON is stored internally, in the library, as a collection of key-value pairs. Like this:

“SampleSection”, null

“SampleSection:Key1”, “Value1”

“SampleSection:Key2”, “Value2”

“SampleSection:Key3”, “Value3”

That means there will be four corresponding rows in SQL-Server, for this JSON. So, if we’re trying to delete the key “SampleSection:Key2” then we’re good only removing the one key-value pair from SQL-Server. On the other hand, if we’re trying to delete the key “SampleSection” then we actually need to remove ALL keys that start with the phrase “SampleSection”. Hence, the LIKE keyword in our SQL.

After the command is created and the SQL parameter is added, the command is executed. The data is now gone from SQL-Server but we also need to remove it from the internal data collection. We do that with a simple LINQ query and a loop that removes the matching keys from that collection.

Using the ADO provider.

We’ll need to create a SQL-Server database in order to use our new ADO provider. To do that, start by creating a new database using Microsoft SQL Server Management Studio. Call the database whatever you like. It doesn’t matter really, just remember what you called it when we go to create a connection string to it.

Here is a bit of SQL to create a table that will hold the configuration related key-value pairs:

Once that’s done, the next step is to add a reference to our new provider, in the REST web application project. To do that, select the CG.Configuration.Host project (or whatever you called it) in the Solution Explorer window, in Visual Studio, then right click and choose the “Add Reference” menu choice.

addreference

That brings up the “Reference Manager” dialog. Make sure the CG.Configuration.Ado project is selected (or whatever you called it), then hit the OK button.

The next change is to the Startup.cs file, in our REST web application project. Here is what the ConfigureServices method looks like, to being with:

And we’ll be changing it to look like this, instead:

Obviously, you’ll need to change the connection string to match your needs.

Now,we need to add some test data to the database. Here are some SQL insert statements to do just that:

Again, you might need to modify these statements if your database, or table name don’t match.

Now, all we need to do, to prove out our new is run the solution in Visual Studio, then go back into POSTMAN and follow the same steps we did before. This time we’ll see the data coming from SQL-Server, instead of the config.json file. Also, when we insert a new value using POSTMAN, we’ll see it reflected in the underlying database.

Testing the Delete can also be done using POSTMAN. Just enter the REST endpoint and select the DELETE Verb, like this:

postmandelete

Then, when you press the Send button in POSTMAN, you’ll be able to see the value is now gone from SQL-Server, and the notification has fired back into the example console application.

Final Thoughts

So in two articles I’ve demonstrated that constructing a fully capable shared configuration service, using .NET Core 2.1, with a REST interface and a SignalR based backchannel for update notifications is not only doable but relatively easy. What’s here isn’t production ready, of course, for that you would want better error checking, perhaps retry logic for the database operations, and, of course, security for the REST application. Still, those are all things that any project would require, in order to be put into production.

I would point out that the REST interface I created for this article needs a few additional method in order to completely cover the capabilities of the Microsoft library. For instance, there is a capability to list child sections, and another capability to fetch a section by key. Both are easy to add to the REST interface, but, I’ll leave both as an exercise for the reader. Also, I usually add a file upload method to the REST interface, so that callers can then supply a set of JSON values all at once. It’s much easier than trying to POST values in one at a time. In fact, what I usually do is work from a JSON file in development, then switch to my configuration service to QA testing and production. When I’m ready, I just upload that JSON file to my REST service and I’m done.

Wrapping Up

I’ll zip my project up and include it as a link in this article. I hope everyone likes the article. I hope everyone gains some benefit from my ideas for a quick shared configuration tool.

Have fun with the code!

 

The project for this article can be downloaded HERE: CG.Configuration.Host