Last time I covered the abstractions and data logic for our nanoservice. This time I’ll cover the data layer, including the database, the repositories, and any logic required to pull everything together.
For this iteration, I chose to implement the database using Microsoft’s SQL Server. At some point I’ll probably come back and craft a no-sql version of the data-layer, since I’d eventually like to host this service online and Azure CosmoDb fees are cheaper than Azure SQL-Server fees. But, for now, we’ll start by looking at the data-context:
public class ObsidianDbContext : DbContext
{
public virtual DbSet<MimeType> MimeTypes { get; set; }
public virtual DbSet<FileExtension> FileExtensions { get; set; }
public ObsidianDbContext(
DbContextOptions<ObsidianDbContext> options
) : base(options)
{
}
protected override void OnModelCreating(
ModelBuilder modelBuilder
)
{
modelBuilder.ApplyConfiguration(new MimeTypeMap());
modelBuilder.ApplyConfiguration(new FileExtensionMap());
base.OnModelCreating(modelBuilder);
}
}
First, notice that I chose to set the visibility of the data-context class to internal. That’s because I consider this class to be an implementation detail of my data layer, and so, I hid it. Of course, doing that means I’m also responsible for ensuring that nothing outside my data layer will ever need to directly interact with my data-context. I’m alright with that, but, it something to keep in mind as we work through things.
The ObsidianDbContext class derives from DbContext, like any Entity Framework or EFCORE database class would. The class contains a DbSet for mime types as well as file extensions. The constructor accepts a standard DbContextOptions class, which is how the caller configures the data-context during creation time. We’ll look at how we create instances of ObsidianDbContext, shortly.
We have overridden the OnModelCreating method, in order to setup the logic model for our data-context. Inside the method, we simple call ModelBuilder.Applyconfiguration twice, passing in a mapping instance as a parameter. Those mapping instances are named MimeTypeMap, and FileExtensionMap. We’ll cover both of those classes, shortly. Once OnModelCreating has finished, our data-context has a complete map between our database and our entity classes.
Model mapping is performed, for the MimeType entity class, using a class named MimeTypeMap. Here is the listing for that class:
internal class MimeTypeMap : AuditedModelMapBase<Models.MimeType>
{
public override void Configure(
EntityTypeBuilder<Models.MimeType> builder
)
{
builder.ToTable("MimeTypes", "Obsidian");
builder.Property(e => e.Id)
.IsRequired()
.ValueGeneratedOnAdd();
builder.HasKey(e => new { e.Id });
builder.Property(e => e.Type)
.IsRequired()
.HasMaxLength(127);
builder.Property(e => e.SubType)
.IsRequired()
.HasMaxLength(127);
builder.Property(e => e.Description)
.IsRequired()
.HasMaxLength(128);
builder.HasMany(e => e.Extensions)
.WithOne()
.HasForeignKey(e => e.MimeTypeId)
.OnDelete(DeleteBehavior.Restrict);
builder.HasIndex(e => new
{
e.Type,
e.SubType,
e.Description
}).IsUnique();
base.Configure(builder);
}
}
From top to bottom, we see that MimeTypeMap derives from a class named AuditedModelMapBase. We’ll cover that class shortly. The class overrides a single method, named Configure. This method is where we’ll buildup the mapping between our database and our entity type.
The first thing we do in this method is call the EntityTypeBuilder.ToTable method, to tell EFCORE what we want our database table to be called. In this case, I chose to name the table “MimeTypes’, and put in inside a schema called “Obsidian”. By using a schema for my tables, I make it easier and safer to combine databases for more than one nanoservice, or even microservice, without much less possibility of name clashes occurring (two tables named Foo, for instance). Next we map the primary key for the table, which in this case, is an identity column named ‘Id’. Then we map the properties for the MimeType class.
Since mime types have an associated with file extensions, the MimeType class has a collection of FileExtension objects, named Extensions. We wire up that relation with the call to:
builder.HasMany(e => e.Extensions)
.WithOne()
.HasForeignKey(e => e.MimeTypeId)
.OnDelete(DeleteBehavior.Restrict);
Which creates the relationship for us, including setting up the foreign key and the cascade rule.
The only thing we do, in this method, is setup the index for the table. For MimeTypes, we create a unique index on the Type, SubType and Description fields.
The mapping for the FileExtension entity type is performed by the FileExtensionMap class. Here’s the listing for that:
public override void Configure(
EntityTypeBuilder<Models.FileExtension> builder
)
{
builder.ToTable("FileExtensions", "Obsidian");
builder.Property(e => e.Id)
.IsRequired()
.ValueGeneratedOnAdd();
builder.Property(e => e.MimeTypeId)
.IsRequired();
builder.HasKey(e => new { e.Id, e.MimeTypeId });
builder.Property(e => e.Extension)
.IsRequired()
.HasMaxLength(260);
builder.HasIndex(e => new
{
e.Extension
}).IsUnique();
base.Configure(builder);
}
This class also overrides the Configure method, to setup a map between our FileExtension entity class and our database. In that method we map to the “FileExtensions” table, in the “Obsidian” schema. The method then sets up the primary key for the table, on the Id property. The other properties are then mapped. Finally, the index for the table is created, ensuring we have unique file extensions in the table.
Both mapping classes, MimeTypeMap and FileExtrensionMap, derive from the AuditedModelMapBase class. That class listing is shown here:
internal abstract class AuditedModelMapBase<T> : IEntityTypeConfiguration<T>
where T : AuditedModelBase
{
public virtual void Configure(
EntityTypeBuilder<T> builder
)
{
builder.Property(e => e.CreatedBy)
.HasMaxLength(50)
.IsRequired();
builder.Property(e => e.CreatedDate)
.IsRequired()
.HasDefaultValue(DateTime.Now);
builder.Property(e => e.UpdatedBy)
.HasMaxLength(50);
builder.Property(e => e.UpdatedDate);
}
}
As we see, this class also overrides the Configure method, and uses it to setup mapping for all the properties of the AuditedModelBase class. That class, which is the base for all the model types in this project, has properties for tracking changes to a model. The AuditedModelBase class is part of the CG.Business NUGET package, which is available HERE.
We’ve covered the data-context, and the mapping classes that support it. Let’s now look through the support code for the data-context. Because we use EFCORE, we need to deal with things like migrations. In order to lessen the pain of working with EFCORE migrations, I added a class named DesignTimeObsidianDbContextFactory. That class implements the IDesignTimeDbContextFactory interface, which is part of EFCORE. This class is how EFCORE can create instances of our internal ObsidianDbContext class, in order to apply migrations, at runtime. Let’s go look at this class:
public class DesignTimeObsidianDbContextFactory : IDesignTimeDbContextFactory<ObsidianDbContext>
{
public ObsidianDbContext CreateDbContext(string[] args)
{
#if DEBUG
var optionsBuilder = new DbContextOptionsBuilder<ObsidianDbContext>();
optionsBuilder.UseSqlServer(
"Server=.;Database=CG.Obsidian.Web;Trusted_Connection=True;MultipleActiveResultSets=true"
);
return new ObsidianDbContext(optionsBuilder.Options);
#else
return null; // never, never, never in production.
#endif
}
}
The class has a CreateDbContext method, which is what creates the data-context instances when EFCORE wants to apply migrations. Inside CreateDbContext, we simple create an instance of our data-context and configure it to use SQL-Server, including passing in a connection string.
You’ve probably noticed a couple things in the method. First, we’ve hard coded the connection string. Second, we’ve used scaffolding to remove the possibility of creating data-context instance in a Release build. Let’s talk about those two things.
The scaffolding is there simply because I’d rather have this code fail, in production (which is the only place I would ever run it in Release mode), than actually apply migrations to my production database. Not that I ever attempt to apply migrations in production. I prefer to generate scripts for the changes, then review the scripts, then apply them in a sandbox, then test – and then apply the scripts as part of my deployment to production. Still, accidents happen, malicious people exists, so I take this precaution here, to make it hard to accidentally mess up my production database.
The connection string is hard coded here for two reasons: First, it ensures that I can never apply migrations to anything other than my development database. This is similar to my reasoning in the paragraph, above. Long story short, I don’t think it should be easy to accidentally apply migrations to the wrong database, so, I hard coded it here. Second reason is, EFCORE doesn’t understand that I’m using the ASP.NET DI container, so, it also doesn’t know how to deal with a non-default constructor on the DesignTimeObsidianDbContextFactory class. By non-default, I mean a constructor with parameters. That means I can’t pass in an options class, or even an IConfiguration object, or even the connection string. Now, I could do other things to go out and read the connection string here, obviously, but, I chose not to do that because I really want this string to be difficult to change.
If you’re new to working with EFCORE, or even just new to working with EFCORE migrations, let me explain that when we go to a command line and execute a statement like:
dotnet ef migrations add InitialCreate --project .\src\Data\CG.Obsidian.SqlServer --context "ObsidianDbContext" --verbose
Which, by the way, is how I created the migrations for this project. Then EFCORE looks into this assembly, and finds the DesignTimeObsidianDbContextFactory class, which it then uses to obtain an instance of our ObsidianDbContext class. EFCORE needs that data-context object in order to do anything related to migrations. Now, if I had chosen to make the ObsidianDbContext class have public visibility, then I wouldn’t have needed the DesignTimeObsidianDbContextFactory class at all. I only need it because nothing outside my assembly can find the ObisidianDbContext type, because I chose to make that type internal. Make sense?
Next we’ll walk through the repository types. There are two: MimeTypeRepository, and FileExtensionRepository. Because the two classes are so similar, I’ll only cover the MimeTypeRepository in this article. Here is the listing for MimeTypeRepository:
public class MimeTypeRepository :
EFCoreRepositoryBase<ObsidianDbContext, IOptions<ObsidianRepositoryOptions>>,
IMimeTypeRepository
{
protected ILogger<MimeTypeRepository> Logger { get; set; }
public MimeTypeRepository(
IOptions<ObsidianRepositoryOptions> options,
DbContextFactory<ObsidianDbContext> factory,
ILogger<MimeTypeRepository> logger
) : base(options, factory)
{
Guard.Instance().ThrowIfNull(logger, nameof(logger));
Logger = logger;
}
public virtual IQueryable<MimeType> AsQueryable()
{
try
{
var context = Factory.Create();
var query = context.MimeTypes
.Include(x => x.Extensions)
.OrderBy(x => x.Type)
.ThenBy(x => x.SubType);
return query;
}
catch (Exception ex)
{
Logger.LogError(
ex,
"Failed to query for mime types!"
);
throw new RepositoryException(
message: $"Failed to query for mime types!",
innerException: ex
).SetCallerInfo()
.SetOriginator(nameof(MimeTypeRepository))
.SetDateTime();
}
}
public virtual async Task<MimeType> AddAsync(
MimeType model,
CancellationToken cancellationToken = default
)
{
try
{
Guard.Instance().ThrowIfNull(model, nameof(model));
var context = Factory.Create();
var entity = await context.MimeTypes.AddAsync(
model,
cancellationToken
).ConfigureAwait(false);
await context.SaveChangesAsync(
cancellationToken
).ConfigureAwait(false);
return entity.Entity;
}
catch (Exception ex)
{
Logger.LogError(
ex,
"Failed to add a new mime type!",
(model != null ? JsonSerializer.Serialize(model) : "null")
);
throw new RepositoryException(
message: $"Failed to add a new mime type!",
innerException: ex
).SetCallerInfo()
.SetOriginator(nameof(MimeTypeRepository))
.SetMethodArguments(("model", model))
.SetDateTime();
}
}
public virtual async Task<MimeType> UpdateAsync(
MimeType model,
CancellationToken cancellationToken = default
)
{
try
{
Guard.Instance().ThrowIfNull(model, nameof(model));
var context = Factory.Create();
var originalModel = context.MimeTypes
.Where(x => x.Id == model.Id)
.Include(x => x.Extensions)
.FirstOrDefault();
if (null == originalModel)
{
throw new KeyNotFoundException(
message: $"Key: {model.Id}"
);
}
originalModel.UpdatedDate = model.UpdatedDate;
originalModel.UpdatedBy = model.UpdatedBy;
originalModel.Type = model.Type;
originalModel.SubType = model.SubType;
originalModel.Description = model.Description;
originalModel.Extensions = model.Extensions;
await context.SaveChangesAsync(
cancellationToken
).ConfigureAwait(false);
return originalModel;
}
catch (Exception ex)
{
Logger.LogError(
ex,
"Failed to update an existing mime type!",
(model != null ? JsonSerializer.Serialize(model) : "null")
);
throw new RepositoryException(
message: $"Failed to update an existing mime type!",
innerException: ex
).SetCallerInfo()
.SetOriginator(nameof(MimeTypeRepository))
.SetMethodArguments(("model", model))
.SetDateTime();
}
}
public virtual async Task DeleteAsync(
MimeType model,
CancellationToken cancellationToken = default
)
{
try
{
Guard.Instance().ThrowIfNull(model, nameof(model));
var context = Factory.Create();
context.MimeTypes.Remove(model);
await context.SaveChangesAsync(
cancellationToken
).ConfigureAwait(false);
}
catch (Exception ex)
{
Logger.LogError(
ex,
"Failed to delete an existing mime type!",
(model != null ? JsonSerializer.Serialize(model) : "null")
);
throw new RepositoryException(
message: $"Failed to delete an existing mime type!",
innerException: ex
).SetCallerInfo()
.SetOriginator(nameof(MimeTypeRepository))
.SetMethodArguments(("model", model))
.SetDateTime();
}
}
}
As we see above, the MimeTypeRepository class derives from the EFCoreRepositoryBase class. That class is part of the CG.Business NUGET package, which can be found HERE. There’s nothing super magical about the EFCoreRepositoryBase, it just contains a way to associate an options type and it contains a property for those options, as well as for a DbContextFactory to create instances of our data-context, at runtime. In case anyone is wondering, I use the DbContextFactory type here, instead of a ObsidianDataContext instance, because I prefer to work with disconnected data, and so, I prefer to create my data-context instances, do some work, then destroy the context. The DBContextFactory type allows me to follow that approach with a little less fuss.
MimeTypeRepository does contain a property for a logger object, which is passed in and initialized in the constructor.
Everything else in the repository is an implementation of the IMimeTypeRepository interface. Let’s start with the AsQueryable method. AsQueryable begins by using the Factory property to access the base class’s DbContextFactory instance. Calling the Create method on that object returns us a data-context instance. Once we have the data-context, we simply access the MimeTypes data set, then call the EFCORE Include extension method, to pull in any associated file extension objects. After doing a little ordering, we return the results as an IQueryable<T> object.
The AddAsync method also starts by creating a data-context instance. Then it calls the AddAsync method on the MimeTypes data set, to add the specified model to the database. Afterwards, we call the SaveChangesAsync on the data-context, to save our changes to the underlying database.
The UpdateAsync method starts by creating a data-context instance. Then it uses LINQ to search for an existing model with the specified primary key. Once again, we also include the associated file extension objects. Finally, we use the LINQ FirstOrDefault method to ensure we either get a valid, matching MimeType reference, or NULL. We take this step here because need a copy of the specified mime type that is connected to, and tracked by, our current data-context instance. Also, we do it to determine whether the specified mime type even exists, before we try to update it.
Assuming everything is fine and we have the required mime type reference, we then update any editable properties, on that reference, using the object that was passed into the method, as an argument. Why do we do that? Well, remember, we did some extra work to get a tracked instance of the mime type object. Since that object is tracked, it means we can defer to EFCORE, for keeping track of what, if anything has changed on the object. So, for instance, if we added a file extension, or took one away, then called this UpdatedAsync method, then thanks to that tracking, EFCORE would be able to work out what needs to be written to the underlying database, in order to synchronize those changes. Finally, we call the SaveChangesAsync on the data-context, to save our changes to the underlying database.
The last method on the repository is called DeleteAsync. This method starts like all the others, by calling the DbContextFactory.Create method, to return a data-context instance. Then we use the Remove method, on the MimeTypes data set, to remove the specified data object from the underlying database. Finally, we call the SaveChangesAsync on the data-context, to save our changes to the underlying database
So at this point we’ve covered the data-context, the factory that allows EFCORE to create instances of our data-context, and one of the two repository classes. What left? Not that much, actually. Like most of the code in the project, this code relies on the ASP.NET DI container to wire all our complicated objects together, at runtime. In order for that to work though, we have to tell the DI contains what our objects are, and how they want to work together. For this data layer assembly, I do that with two extensions classes, one hung off the IServiceCollection type, and the other hung off the IApplicationBuilder type. Let’s go look at those now.
The ServiceCollectionExtensions class looks like this:
public static partial class ServiceCollectionExtensions
{
public static IServiceCollection AddSqlServerRepositories(
this IServiceCollection serviceCollection,
IConfiguration configuration,
ServiceLifetime serviceLifetime = ServiceLifetime.Scoped
)
{
Guard.Instance().ThrowIfNull(serviceCollection, nameof(serviceCollection))
.ThrowIfNull(configuration, nameof(configuration));
serviceCollection.ConfigureOptions<ObsidianRepositoryOptions>(
configuration,
out var repositoryOptions
);
serviceCollection.AddTransient<ObsidianDbContext>(serviceProvider =>
{
var options = serviceProvider.GetRequiredService<IOptions<ObsidianRepositoryOptions>>();
var builder = new DbContextOptionsBuilder<ObsidianDbContext>();
if (options.Value.LogAllQueries)
{
builder.UseLoggerFactory(
serviceProvider.GetRequiredService<ILoggerFactory>()
);
}
#if DEBUG
builder.EnableDetailedErrors()
.EnableSensitiveDataLogging();
#endif
builder.UseSqlServer(options.Value.ConnectionString);
var context = new ObsidianDbContext(builder.Options);
return context;
});
serviceCollection.Add<DbContextFactory<ObsidianDbContext>>(serviceLifetime);
serviceCollection.Add<IMimeTypeRepository, MimeTypeRepository>(serviceLifetime);
serviceCollection.Add<IFileExtensionRepository, FileExtensionRepository>(serviceLifetime);
return serviceCollection;
}
}
This class has a single method, AddSqlServerRepositories, which is used to register our types with the DI container. As shown above, that method starts by calling an extension method off the IConfiguration type called ConfigureOptions. That method is part of my CG.Options NUGET package, which is available HERE. The method itself is beyond the scope of this article. For now, just know that it populates an instance of ObsidianRepositoryOptions, unprotects any protected properties, validates the results, then registers those options with the DI container, as a singleton service.
The next thing the method does is register the ObsidianDbContext type. We register the type as transient, since we will create temporary instances using the DbContextFactory type, in our repositories. Inside the factory delegate, we see that we pull up an instance of the ObsidianRepositoryOptions object that we registered previously. We then create a standard EFCORE DbContextOptionsBuilder instance. Once we have the EFCORE object instance, we configure it to use SQLServer, with our connection string, and we also, optionally, enable query logging and detailed error logging. Those last two are helpful in development but would probably be turned off, in production. Once the EFCORE options are completely configured, we use it to create the instance of our data-context class, ObsidianDbContext. Then we return the instance.
After registering the factory for our data-context, we then register the DbContextFactory type, along with the two repository classes: MimeTypeRepository and FileExtensionRepository.
The next extension method is in the ApplicationBuilderExtensions class. That looks like this:
public static partial class ApplicationBuilderExtensions
{
public static IApplicationBuilder UseSqlServerRepositories(
this IApplicationBuilder applicationBuilder,
string configurationSection
)
{
Guard.Instance().ThrowIfNull(applicationBuilder, nameof(applicationBuilder))
.ThrowIfNullOrEmpty(configurationSection, nameof(configurationSection));
applicationBuilder.UseEFCore<ObsidianDbContext, ObsidianRepositoryOptions>(
(context, wasDropped, wasMigrated) =>
{
var logger = applicationBuilder.ApplicationServices.GetRequiredService<ILogger<ObsidianDbContext>>();
logger.LogInformation(
$"Initialized: '{context.ContextId}', dropped: '{wasDropped}', migrated: '{wasMigrated}'"
);
context.ApplySeedData(
logger
);
});
return applicationBuilder;
}
}
This method does little more than call my UseEFCore extension method. That method, UseEFCore, is part of the CG.Linq.EFCore NUGET package, which is available HERE. The internals of the UseEFCore method are beyond the scope of this article. For now, just understand that this method looks at several flags in the associated options, and uses those flags to determine whether to drop the underlying database, create a new database, apply migrations, or even seed the new database. When dealing with EFCORE databases (at least for SqlServer) these steps are ones that I’ve needed to perform over and over. So, I encapsulated all that logic into the UseEFCore method, shown here, rather than continue to write the same code, ad nauseum. What I’ve added here, in the UseSqlServerRepositories method, beyond simply calling UseEFCore, is the line where I use a data-context instance, that was created by the UseEFCore method, to seed the underlying database with startup data. That is the call to the ApplySeedData extension method, which is hung off the ObsidianDataContext type.
The ApplySeedData method looks like this:
internal static partial class DbContextExtensions
{
public static void ApplySeedData(
this ObsidianDbContext context,
ILogger<ObsidianDbContext> logger
)
{
Guard.Instance().ThrowIfNull(context, nameof(context));
logger.LogInformation($"Starting seeding operations on: {context.ContextId}");
context.SeedMimeTypes(logger);
context.SeedFileExtensions(logger);
logger.LogInformation($"Finished seeding operations on: {context.ContextId}");
}
private static void SeedMimeTypes(
this ObsidianDbContext context,
ILogger<ObsidianDbContext> logger
)
{
if (true == context.MimeTypes.Any())
{
logger.LogInformation($"Skipping seeding operation on: [Obsidian].[MimeTypes].");
return;
}
logger.LogInformation($"Starting seeding operation on: [Obsidian].[MimeTypes]. This could take a bit ...");
int docs = 0;
long errors = 0;
long skipped = 0;
long rows = 0;
using (var handler = new HttpClientHandler())
{
handler.UseDefaultCredentials = true;
handler.UseCookies = true;
using (var client = new HttpClient(handler))
{
client.DefaultRequestHeaders.UserAgent.ParseAdd(
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
);
client.BaseAddress = new Uri("http://www.iana.org/assignments/media-types/");
var endpoints = new string[]
{
"application.csv",
"audio.csv",
"font.csv",
"image.csv",
"message.csv",
"model.csv",
"multipart.csv",
"text.csv",
"video.csv"
};
foreach (var endpoint in endpoints)
{
try
{
logger.LogInformation($"Downloading: {endpoint} ...");
var csv = client.GetStringAsync(endpoint).Result;
if (string.IsNullOrEmpty(csv))
{
errors++;
continue; // Nothing left to do.
}
try
{
var lines = csv.Split(Environment.NewLine);
foreach (var line in lines.Skip(1))
{
var fields = line.Split(',');
if (3 != fields.Length)
{
skipped++;
continue; // Nothing left to do.
}
var parts = new string[0];
if (fields[1].Trim() == "")
{
parts = new string[]
{
Path.GetFileNameWithoutExtension(endpoint),
fields[0]
};
}
else
{
parts = fields[1].Split('/');
}
if (2 != parts.Length)
{
skipped++;
continue; // Nothing left to do.
}
if (context.MimeTypes.Any(x =>
x.Type == parts[0] && x.SubType == parts[1]))
{
skipped++;
continue; // Nothing left to do.
}
context.Add(new Models.MimeType
{
CreatedBy = "seed",
CreatedDate = DateTime.Now,
Type = parts[0],
SubType = parts[1],
Description = fields[0]
});
context.SaveChanges();
rows++;
}
}
catch (Exception ex)
{
errors++;
logger.LogError($"Failed to process: {endpoint}", ex);
}
}
catch (Exception ex)
{
errors++;
logger.LogError($"Failed to download: {endpoint}", ex);
}
docs++;
}
}
}
logger.LogInformation(
$"Finished seeding [Obsidian].[MimeTypes]. " +
$"docs: {docs}, rows: {rows}, skipped: {skipped}, errors: {errors}"
);
}
private static void SeedFileExtensions(
this ObsidianDbContext context,
ILogger<ObsidianDbContext> logger
)
{
if (true == context.FileExtensions.Any())
{
return;
}
var cannedData = new (string, string, string)[]
{
new (".aac", "audio", "aac"),
new (".abw", "application", "x-abiword"),
new (".arc", "application", "x-freearc"),
new (".avi", "video", "x-msvideo"),
new (".azw", "application", "vnd.amazon.ebook"),
new (".bin", "application", "octet-stream"),
new (".bmp", "image", "bmp"),
new (".bz", "application", "x-bzip"),
new (".bz2", "application", "x-bzip2"),
new (".csh", "application", "x-csh"),
new (".css", "text", "css"),
new (".csv", "text", "csv"),
new (".doc", "application", "vnd.openxmlformats-officedocument.wordprocessingml.document"),
new (".eot", "application", "vnd.ms-fontobject"),
new (".epub", "application", "epub+zip"),
new (".gz", "application", "gzip"),
new (".gif", "image", "gif"),
new (".htm", "text", "html"),
new (".html", "text", "html"),
new (".ico", "image", "vnd.microsoft.icon"),
new (".ics", "text", "calendar"),
new (".jar", "application", "java-archive"),
new (".jpg", "image", "jpeg"),
new (".jpeg", "image", "jpeg"),
new (".js", "text", "javascript"),
new (".json", "application", "json"),
new (".jsonld", "application", "ld+json"),
new (".mid", "audio", "midi"),
new (".midi", "audio", "x-midi"),
new (".mjs", "text", "javascript"),
new (".mp3", "audio", "mpeg"),
new (".cda", "audio", "x-cdf"),
new (".mp4", "video", "mp4"),
new (".mpeg", "video", "mpeg"),
new (".mpkg", "application", "vnd.apple.installer+xml"),
new (".odp", "application", "vnd.oasis.opendocument.presentation"),
new (".ods", "application", "vnd.oasis.opendocument.spreadsheet"),
new (".odt", "application", "vnd.oasis.opendocument.text"),
new (".oga", "audio", "ogg"),
new (".ogv", "video", "ogg"),
new (".ogx", "application", "ogg"),
new (".opus", "audio", "opus"),
new (".otf", "font", "otc"),
new (".png", "image", "png"),
new (".pdf", "application", "pdf"),
new (".php", "application", "x-httpd-php"),
new (".ppt", "application", "vnd.ms-powerpoint"),
new (".pptx", "application", "vnd.openxmlformats-officedocument.presentationml.presentation"),
new (".rar", "application", "vnd.rar"),
new (".rtf", "application", "rtf"),
new (".sh", "application", "x-sh"),
new (".svg", "image", "svg+xml"),
new (".swf", "application", "x-shockwave-flash"),
new (".tar", "application", "x-tar"),
new (".tif", "image", "tiff"),
new (".tiff", "image", "tiff"),
new (".ts", "video", "ts"),
new (".ttf", "font", "ttf"),
new (".text", "text", "plain"),
new (".vsd", "application", "vnd.visio"),
new (".wav", "audio", "wav"),
new (".weba", "audio", "webm"),
new (".webm", "audio", "webm"),
new (".webp", "audio", "webp"),
new (".woff", "font", "woff"),
new (".woff2", "font", "woff2"),
new (".xhtml", "application", "xhtml+xml"),
new (".xls", "application", "vnd.ms-excel"),
new (".xlsx", "application", "vnd.openxmlformats-officedocument.spreadsheetml.sheet"),
new (".xml", "text", "xml"),
new (".xul", "application", "vnd.mozilla.xul+xml"),
new (".zip", "application", "zip"),
new (".3gp", "video", "3gpp"),
new (".3g2", "video", "3gpp2"),
new (".7z", "application", "x-7z-compressed")
};
logger.LogInformation($"Starting seeding operation on: [Obsidian].[FileExtensions].");
long skipped = 0;
long rows = 0;
long errors = 0;
try
{
foreach (var row in cannedData)
{
if (context.FileExtensions.Any(x =>
x.Extension == row.Item1
))
{
skipped++;
continue; // Nothing left to do.
}
var mimeType = context.MimeTypes.FirstOrDefault(x =>
x.Type == row.Item2 && x.SubType == row.Item3
);
if (null == mimeType)
{
skipped++;
continue; // Nothing left to do.
}
context.Add(new Models.FileExtension()
{
CreatedBy = "seed",
CreatedDate = DateTime.Now,
Extension = row.Item1,
MimeTypeId = mimeType.Id
});
rows += context.SaveChanges();
}
}
catch (Exception ex)
{
errors++;
}
logger.LogInformation(
$"Finished seeding [Obsidian].[FileExtensions]. " +
$"rows: {rows}, skipped: {skipped}, errors: {errors}"
);
}
}
Sorry for this listing, it’s a bit long and messy. I’ll be refactoring much of this code, at some point, since I’ll eventually turn it into a job to periodically check for new or outdated mime types.. For now though, it’s just this code to seed an empty database.
The ApplySeedData method is what is called during the application startup, inside the UseSqlServerRepositories method. ApplySeedData, in turn, calls two private methods: SeedMimeTypes and SeedFileExtensions. Any time the nanoservice is run against an otherwise empty database, this seed logic is called when the service starts. That means, on first startup, or any time the configuration is set to drop the database on startup. Of course, none of this ever happens in Release mode, since it would be crazy to write code that might, accidentally or on purpose, dump seed data into a production database …
SeedMimeTypes creates an HTTP client object, then it uses that client to access the publicly available documents on the www.iana.org website. The variable endpoints contains an array of document names, where each document contains a different classification of mime type. The code loops through the endpoints array, downloading and processing each type of document.
Each time a document is downloaded, it is immediately split into lines (the documents themselves are CSV files). Each line is then processed in an inner loop. We split the fields out into another array, then do some simple “fixing” of the data, to be sure it’s something we can process. Once we’ve done that, we check to see if we’ve already added this mime type. If not, we create a MimeType model, and add it to the data-context. Finally, we call SaveChanges, on the data-context, to save our work.
The SeedFileExtensions method works a little differently. Since I haven’t yet found a document containing file extensions for mime types, I decided to get things started with a few files types that I know about and frequently use. If I find a better online source for this information, I’ll use it in place of this code.
The variable cannedData contains an array of tuples contain the file extension, the mime type, and the mime subtype. Looping over the cannedData collection, we first look to make sure we don’t already have this file extension in the database. Assuming we don’t, we then find the associated MimeType, from the database. Then we create a FileExtension model and add the record to the data-context. Finally, we call SaveChanges, on the data-context, to save our work.
Were’ almost done with the data layer at this point. The only thing left to discuss is how we tell the ASP.NET DI container how all these pieces fit together. That starts with the standard Blazor Startup class. Here is an abbreviated listing of our Startup class:
public class Startup
{
public IConfiguration Configuration { get; }
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public void ConfigureServices(IServiceCollection services)
{
services.AddCustomObsidian(
Configuration.GetSection("CG.Obsidian")
);
}
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
app.UseCustomObsidian(env);
}
}
I’ve taken all the unrelated bits of the Startup class out, for clarity. We see that, in the ConfigureServices method, we call an extension method named AddCustomObsidian, passing in the configuration section “CG.Obsidian”. That call takes care of registering all the various types with the DI container, but, it doesn’t handle any startup logic. For that, we call the UseCustomObsidian method, from inside the Startup class’s Configure method. Let’s go look at those last two methods now:
public static partial class ServiceCollectionExtensions
{
public static IServiceCollection AddCustomObsidian(
this IServiceCollection serviceCollection,
IConfiguration configuration
)
{
Guard.Instance().ThrowIfNull(serviceCollection, nameof(serviceCollection))
.ThrowIfNull(configuration, nameof(configuration));
serviceCollection.AddObsidianStores(
configuration.GetSection("Stores"),
ServiceLifetime.Singleton
);
serviceCollection.AddObsidianManagers(
configuration.GetSection("Managers"),
ServiceLifetime.Singleton
);
serviceCollection.AddRepositories(
configuration.GetSection("Repositories"),
ServiceLifetime.Singleton
);
return serviceCollection;
}
}
At this point, this code should look somewhat familiar. This is where we call the extension methods that we walked through, earlier in this article. AddObidianStores, AddObsidianManagers, and … wait, AddRepositories?? What happened to AddSqlServerRepositories?? Well, in order to avoid hard coding a dependency on EFCORE, within our nanoservice, I have used the AddRepositories method here, in place of the one we covered earlier, AddSqlServerRepositories. The workings of the AddReposititories method is beyond the scope of this article, so, just realize that it uses the configuration data we passed in, from the “CG.Obsidian:Repositories” section, and dynamically locates and loads the CG.Obsidian.SqlServer assembly. After that, it dynamically locates and calls the AddSqlServerRepositories method, on our behalf. That way, we aren’t hard coded to use SQL-Server. We could, if we wanted to, simply change our configuration and startup with a completely different data layer – with no code changes, or testing, or redeploys.
Here is the listing for the final method we’ll cover in this article:
public static partial class ApplicationBuilderExtensions
{
public static IApplicationBuilder UseCustomObsidian(
this IApplicationBuilder applicationBuilder,
IWebHostEnvironment webHostEnvironment
)
{
Guard.Instance().ThrowIfNull(applicationBuilder, nameof(applicationBuilder))
.ThrowIfNull(webHostEnvironment, nameof(webHostEnvironment));
applicationBuilder.UseRepositories(
"CG.Obsidian:Repositories"
);
return applicationBuilder;
}
}
Here we see that, inside of UseCustomObsidian method, we call the UseRepositories extension method. That method works much like the AddRepositories method we just covered. It also dynamically locates and loads the CG.Obsidian.SqlServer assembly, and then dynamically locates and loads the UseSqlServerReposities extension method that we covered earlier in this article. The big difference here is that, this method assumes that most (if not all) of the services and types have been registered with the ASP.NET DI container. So, this method instead focuses on running any startup logic needed for the data layer. Recall that, inside our UseSqlServerRepositories method, we added code to do things like create databases, drop databases, apply migrations, and seed data.
This is how everything gets pulled together, in the nanoservice, using the ASP.NET DI container. As a result, by the time the nanoservice is up and running, there will be a registered service, in the DI container, for the IMimeTypeManager type. Instances of IMimeTypeManager will contain instances of our stores, which will contain instances of our repositories, which will be capable of creating instances of our data-context connected to our SQL-Server database. Everything will work together and all you, as a developer, will ever need to worry about, is the interface for the IMimeTypeManager type.
The only thing left to cover is the client for the nanoservice. I’ll cover that next time.
