Code Penguin

Custom Polyglot Notebook Kernel

2023-08-05T21:00:00+00:00

Polyglot Notebooks are a great Visual Studio Code extension that allows you to create interactive Notebooks using multiple languages.

Additional language support, extra commands, formatters, etc. can be added using Dotnet Nuget packages.

For this article, we will be creating a custom kernel that will execute basic calculator commands with variable support.

Calculator Kernel and Infrastructure

We will start by creating a new CSharp Project that will contain all of our extension classes:

dotnet new classlib -n PolyglotCalculatorKernel

The kernel we create will be a subkernel to the CSharp Kernel so that it can share variables with other kernels.

We need to add the Microsoft.DotNet.Interactive package that will handle all of the kernel interactions that we require and add a new class for our kernel:

dotnet add package Microsoft.DotNet.Interactive --prerelease

Edit PolyglotCalculatorKernel.csproj and add the following section:

<ItemGroup>
    <None Include="extension.dib" Pack="true" PackagePath="interactive-extensions/dotnet" />
</ItemGroup>

Rename Class1.cs to CalculatorKernel.cs and rename the class to CalculatorKernel as well. CalculatorKernel needs to inherit from Microsoft.DotNet.Interactive.Kernel and needs a constructor that calls the base constructor as well.

using Microsoft.DotNet.Interactive;

namespace PolyglotCalculatorKernel;

public class CalculatorKernel : Kernel
{
    public CalculatorKernel(string name) : base(name)
    {
    }
}

Polyglot Notebook extensions are usually distributed as a Nuget package that contains an extension.dib file that provides the entry point for the Notebook to load the extension.

To make things easy to setup, we will create a NotebookExtensions class with a static Load method that will be called by our extensions.dib file. This keeps all of the code in C# so it is fully compiled.

dotnet new class --name NotebookExtensions

using Microsoft.DotNet.Interactive;

namespace PolyglotCalculatorKernel;

public class NotebookExtensions
{
    public static void Load(Kernel kernel)
    {
        const string kernelName = "calculator";
        if (kernel.RootKernel is not CompositeKernel compositeKernel) return;
        if (compositeKernel.FindKernelByName(kernelName) != null) return;

        var calculatorKernel = new CalculatorKernel(kernelName);
        compositeKernel.Add(calculatorKernel);

        KernelInvocationContextExtensions.Display(KernelInvocationContext.Current, "Calculator has been loaded.", "text/plain");
    }
}

The NotebookExtensions.Load method checks to see if the Calculator Kernel has already been loaded. If not loaded a new instance of the CalculatorKernel class which is then added to the CSharpKernel as a sub kernel. Finally, we display a message so the user knows everything loaded successfully.

Create an extensions.dib file in the project directory with the following contents:

#!csharp

PolyglotCalculatorKernel.NotebookExtensions.Load(Microsoft.DotNet.Interactive.KernelInvocationContext.Current.HandlingKernel.RootKernel);

This just calls our NotebookExtensions.Load method passing in the current root kernel from the Notebook.
If we want to add other kernels or extensions in the future, we can just update the Load method.

Calculator Class

We need a class that will handle of the actual calculator functionality. Since our calculator class is not the main focus of this article, it is going to be extremely basic and will not handle things the way you would in production such as error handling, operator precedence, etc. We just need it to work enough that we can wrap it in a Polyglot Notebook Kernel.

The calculator commands must follow the following syntax rules:

Each number, operator, variable must be separated by a single space.
Numbers must be integer whole numbers.
Anything that can be parsed into a number is considered a number.
The following operators are supported: + (Add), - (Subtract), * (Multiply), = (Assignment).
Variables are created when a value is assigned.

Add a new Calculator class with the following contents:

public class Calculator
{
    public Dictionary<string, int> Variables { get; } = new();
    private readonly List<string> _data = new();

    public int Execute(string code)
    {
        _data.Clear();
        _data.AddRange(code.Split(' '));
        int result;
        if (_data.Count == 0) return 0;
        if (_data.Count > 2 && _data[1] == "=")
        {
            var name = _data[0];
            result = Evaluate(2);
            Variables[name] = result;
        }
        else
        {
            result = Evaluate(0);
        }
        return result;
    }

    private int Evaluate(int startIndex)
    {
        var index = startIndex;
        var result = GetInt(ref index);
        while (index < _data.Count - 1)
        {
            var op = _data[index++];
            result = op switch
            {
                "+" => result + GetInt(ref index),
                "-" => result - GetInt(ref index),
                "*" => result * GetInt(ref index),
                _ => throw new InvalidOperationException($"Unknown operator: {op}")
            };
        }
        return result;
    }

    private int GetInt(ref int index)
    {
        var item = _data[index++];
        return int.TryParse(item, out var value)
            ? value
            : GetVariable(item);
    }

    private int GetVariable(string name)
    {
        return Variables.TryGetValue(name, out var value)
            ? value
            : throw new InvalidOperationException($"Unknown Variable: {name}");
    }
}

If we actually used the class right now in a CSharp Polyglot Notebook cell, it would be able to handle our basic operations:

var calc = new Calculator();
calc.Execute("1 + 2 * 3 - 4").Display(); // Returns 5
calc.Execute("Value = 1 + 2 * 3").Display(); // Returns 9
calc.Execute("Value + 1").Display(); // Returns 10
calc.Execute("Bad").Display(); // Throws an error

Handling the SubmitCode command

Kernels can inherit from various interfaces to indicate that they support commands such as executing code, gathering variables, etc.

We will focus on executing code first so we will inherit from IKernelCommandHandler<SubmitCode> and implement the interface:

public class CalculatorKernel :
    Kernel,
    IKernelCommandHandler<SubmitCode>
{
    private readonly Calculator _calculator = new();

    public CalculatorKernel() : base("Calculator")
    {
    }

    Task IKernelCommandHandler<SubmitCode>.HandleAsync(SubmitCode command, KernelInvocationContext context)
    {
        context.Publish(new CodeSubmissionReceived(command));
        context.Publish(new CompleteCodeSubmissionReceived(command));

        try
        {
            int result = _calculator.Execute(command.Code);

            var formattedValues = FormattedValue.CreateManyFromObject(result);
            context.Publish(new ReturnValueProduced(result, command, formattedValues));
        }
        catch (InvalidOperationException ex)
        {
            context.DisplayStandardError(ex.Message);
        }
        catch (Exception ex)
        {
            context.Fail(command, ex);
        }

        return Task.CompletedTask;
    }
}

We added a private field containing a new instance of our Calculator class that will represent the state of our kernel.

Whenever code is submitted, the HandleAsync method will be called with a SubmitCode instance. SubmitCode.Code contains the actual text of the code.

We publish CodeSubmissionReceived and CompleteCodeSubmissionReceived messages so that the Notebook knows we received the message and will start processing it.

We then send the code we received to our calculator and get the result. The result is then formatted and published back to the Notebook.

Any InvalidOperationExceptions are displayed as Standard Error and any other exceptions are sent back as failures.

Testing the Calculator Kernel

We can now build and test out our calculator in a Notebook by adding a cell that loads our assembly and calls NotebookExtensions.Load. Create a new Polyglot Notebook in the project directory with the following CSharp cell contents:

#r "./PolyglotCalculatorKernel/bin/Debug/net7.0/PolyglotCalculatorKernel.dll"
PolyglotCalculatorKernel.NotebookExtensions.Load(Microsoft.DotNet.Interactive.KernelInvocationContext.Current.HandlingKernel.RootKernel);

We are loading the assembly directly for rapid testing but in the future we will use a Nuget package instead. If you want to make changes, you must click “Restart” on the Polyglot Notebook so that the kernel is unloaded before you try to build it again.

Execute the cell and then create and execute a new cell that uses the new “calculator” kernel with the following contents:

1 + 2 * 3 - 4

You should see it output “5”.

Variables

Our calculator already supports internal variables but with just a couple changes we can support external variables and expose our internal variables to other kernels.

In the Notebook, open the Visual Studio Code Command Pallete (Usually Ctrl + Shift + P on Windows or Command + Shift + P on Mac) and select “Polyglot Notebooks: Focus on Variables view”. This will open a view that displays all of the variables that are available in the Notebook.

Accessing External Variables

Lets create a new calculator cell with the following contents:

#!set --name x --value 5
x + 1

This exposes a new variable called x with a value of 5 that our kernel can use. If you execute this cell now, you’ll get the following errors in the Notebook:

Error: Unrecognized command or argument '#!set'.
Unrecognized command or argument '--name'.
Unrecognized command or argument 'x'.
Unrecognized command or argument '--value'.
Unrecognized command or argument '5'.

To get this to work we need to enable Variable Sharing by updating NotebookExtensions.Load to call the UseValueSharing extension method on our kernel instance:

var calculatorKernel = new CalculatorKernel(kernelName);
calculatorKernel.UseValueSharing();
compositeKernel.Add(calculatorKernel);

Execute the cell again and you get the following error:

Error: Microsoft.DotNet.Interactive.CommandNotSupportedException: Kernel PolyglotCalculatorKernel.CalculatorKernel: calculator (kernel://pid-31897/Calculator) does not support command type SendValue.

This tells us that CalculatorKernel needs to support the SendValue command.

Update CalculatorKernel so that it also inherits from IKernelCommandHandler<SendValue> and implement the interface with the following method:

async Task IKernelCommandHandler<SendValue>.HandleAsync(SendValue command, KernelInvocationContext context)
{
    await SetValueAsync(command, context, (name, value, declaredType) =>
    {
        if (value is int intValue || int.TryParse(value.ToString(), out intValue))
        {
            _calculator.Variables[name] = intValue;
        }
        return Task.CompletedTask;
    });
}

The SetValueAsync method comes from Microsoft.DotNet.Interactive.Kernel and handles some basic conversions for us. It calls a delegate that we provide to actually set the variables. If the variable is an int or can be parsed to an int from a string value then it will be added to our calculator.

If we click “Restart” in the Notebook, build the project and then re-run all of the cells, the last cell should now process as intended and outputs 6.

We can even take variables from other kernels such as CSharp. Create a new cell with the following contents:

var csharpInt = 11;

Execute this cell and the variable view will have a new value displayed. We can use this csharpInt variable in our calculator by using the #!set command. Create a new calculator cell with the following contents and the output should be 20:

#!set --name csharpInt --value @csharp:csharpInt
csharpInt + 9

Exposing Internal Variables

Now that we can get variables into our calculator, we now need to get variables out of our calculator.

Create a new calculator cell with the following contents which will create a new variable calcInt and set its value to 23:

calcInt = 23

Notice how the variables view still only contains the csharpInt value.

Update CalculatorKernel so that it also inherits from IKernelCommandHandler<RequestValueInfos> and implement the interface with the following method:

Task IKernelCommandHandler<RequestValueInfos>.HandleAsync(RequestValueInfos command, KernelInvocationContext context)
{
    var variables = _calculator.Variables
        .Select(v =>
        {
            var formattedValues = FormattedValue.CreateSingleFromObject(v.Value, command.MimeType);
            return new KernelValueInfo(v.Key, formattedValues, typeof(int));
        }).ToArray();
    context.Publish(new ValueInfosProduced(variables, command));
    return Task.CompletedTask;
}

This HandleAsync is given a RequestValueInfos command which contains a MimeType indicating what format the variables are being requested in. We loop through each variable in our calculator and pass the value into FormattedValue.CreateSingleFromObject which will convert the value to the specified MimeType. Finally, the list of variables is then published as a ValueInfosProduced message back to the Notebook.

Click “Restart” in the Notebook, rebuild the project and then click “Run All” in the Notebook. The variable view should now have a few more items such as calcInt.

Let’s use the calcInt in a CSharp kernel. Create a new CSharp cell with the following contents:

#!set --value @calculator:calcInt --name calcInt
Console.WriteLine(calcInt);

This exposes the calcInt variable from our calculator kernel and then writes out the value. If you execute this cell now you will get the following error:

Error: Microsoft.DotNet.Interactive.NoSuitableKernelException: No kernel found for Microsoft.DotNet.Interactive.Commands.RequestValue with target kernel 'calculator'.

This is because we handled the RequestValueInfos command to expose a list of variables but we haven’t handled the RequestValue command to retrieve an individual value.

Update CalculatorKernel so that it also inherits from IKernelCommandHandler<RequestValue> and implement the interface with the following method:

Task IKernelCommandHandler<RequestValue>.HandleAsync(RequestValue command, KernelInvocationContext context)
{
    if (_calculator.Variables.TryGetValue(command.Name, out var value))
    {
        context.PublishValueProduced(command, value);
    }
    else
    {
        context.Fail(command, message: $"Value '{command.Name}' not found in kernel {Name}");
    }
    return Task.CompletedTask;
}

This HandleAsync is given a RequestValue instance that has a Name property indicating what variable is being requested. We attempt to retrieve the variable out of the calculator’s list of variables and then publish the value back to the Notebook.

Click “Restart” in the Notebook, rebuild the project and then click “Run All” in the Notebook. The last cell should now execute correctly and output the value 23.

Auto Completion

Since the calculator kernel keeps track of all of the variables that are defined, we should be able to get a list of them from our Notebook using Auto Complete (Ctrl + Space). We just need to handle the RequestCompletions kernel command.

The RequestCompletions command contains the Code that is being typed and the LinePosition indicating where in the code the completion is being requested for. You can change the list of completions based on the position of the cursor within the code.

First we need to add the following method to our Calculator class so we can get the token at the cursor position:

public static string GetTokenAtPosition(string code, int position, bool partialToken = false)
{
    if (string.IsNullOrEmpty(code)) return string.Empty;
    var endPosition = partialToken ? position : code.IndexOf(' ', position + 1);
    if (endPosition == -1) endPosition = code.Length;
    var codeBeforeCursor = code[..endPosition];
    var tokensBeforeCursor = codeBeforeCursor.Split(' ');
    return tokensBeforeCursor[^1];
}

Update CalculatorKernel so that it also inherits from IKernelCommandHandler<RequestCompletions> and implement the interface with the following method:

Task IKernelCommandHandler<RequestCompletions>.HandleAsync(RequestCompletions command, KernelInvocationContext context)
{
    var token = Calculator.GetTokenAtPosition(command.Code, command.LinePosition.Character, partialToken: true);
    var completionList  = _calculator.Variables
        .Where(v => string.IsNullOrEmpty(token) || v.Key.StartsWith(token))
        .Select(v =>
        {
            return new CompletionItem(v.Key, typeof(int).FullName,
                insertText: string.IsNullOrEmpty(token) ? v.Key : v.Key[token.Length..],
                documentation: $"Current Value = {v.Value}");
        });
    context.Publish(new CompletionsProduced(completionList, command));
    return Task.CompletedTask;
}

We first get the token at the current cursor position. If we find it, we will get a list of all variables that start with that token. If there isn’t a token at the cursor position then we get a list of all variables. We then convert our list of variables into a list of CompletionItem instances and then return a CompletionsProduced instance that includes our converted items.

The CodeCompletionItem.Documentation can be set to give extra information about the item. Since we are just returning a list of integer variables, we will show the current value of the variable which will be displayed in a hover-over window.

We also set the CodeCompletionItem.InsertText based on the token that we are replacing. This lets us only insert the missing part of the selected token instead of putting the entire completed value at the cursor position which would repeat what was already typed.

Hover Text

When you mouse over the text in a cell, we can provide some hover text that can be displayed. For our calculator, it would be great if hovering over a variable would show the current value. We just need to handle the RequestHoverText kernel command.

The RequestHoverText command contains the Code and LinePosition indicating where in the code the the mouse is currently hovering over.

Update CalculatorKernel so that it also inherits from IKernelCommandHandler<RequestHoverText> and implement the interface with the following method:

Task IKernelCommandHandler<RequestHoverText>.HandleAsync(RequestHoverText command, KernelInvocationContext context)
{
    var token = Calculator.GetTokenAtPosition(command.Code, command.LinePosition.Character, partialToken: false);
    if (!string.IsNullOrWhiteSpace(token) && _calculator.Variables.TryGetValue(token, out var value))
    {
        context.Publish(
            new HoverTextProduced(
                command,
                new[] { new FormattedValue("text/markdown", $"**{token}** (Current Value = {value})") },
                new LinePositionSpan(command.LinePosition, command.LinePosition)));
    }
    return Task.CompletedTask;
}

We reuse our Calculator.GetTokenAtPosition but pass in false for partialToken so we get the full token at the specified position.

If the token is one of our variables we then publish a HoverTextProduced instance with a bit of Markdown formatted text that will be displayed in the hover text.

Nuget Packages

So far we have been manually compiling and including our custom kernel as an assembly using the #r "./PolyglotCalculatorKernel/bin/Debug/net7.0/PolyglotCalculatorKernel.dll" magic command. For final distribution, custom kernels should be packaged as Nuget Packages and loaded using something like #r "nuget:CodePenguin.DotNet.Interactive.Calculator,*-*". The ClockExtension Sample walks through the steps to build things as a Nuget package.

The essential steps are adding a few extra properties to the CSPROJ to identify the package:

  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IncludeBuildOutput>true</IncludeBuildOutput>
    <IsPackable>true</IsPackable>
    <PackageDescription>Polyglot Calculator Kernel - Custom Kernel Example</PackageDescription>
  </PropertyGroup>

This indicates that the project is packable, includes the build output in the package and gives a description for the package.

You can now execute the following to build and package the Nuget package:

dotnet build
dotnet pack /p:PackageVersion=0.0.1-beta

You have to specify a version for the Nuget package. Here we are specifying a beta suffix since the Dotnet Interactive assemblies are still in beta. This isn’t required but the dotnet pack command will give you a warning if you don’t.

To test out our Nuget package locally, add a CSharp cell with the following contents:

var debugOutputFolder = new System.IO.DirectoryInfo(@"./bin/Debug/net7.0/").FullName;
var nugetSource = $"nuget:{debugOutputFolder}";

This figures out the Nuget package path for the project since the #i command cannot use relative paths at the moment.

Now add a separate CSharp cell with the following contents:

#i @csharp:nugetSource
#r "nuget:PolyglotCalculatorKernel,*-*"

Now you can execute these two cells to load the Calculator Kernel from local Nuget packages. To distribute the kernel to others, you would need to upload it to https://nuget.org and then you can reference it with just the #r command:

#r "nuget:PolyglotCalculatorKernel,*-*"

The *-* says to load the largest suffixed version it can find. If you execute the above it will find the version I uploaded to Nuget for this article.

Conclusion

Through a feature rich API, custom Polyglot Notebook Kernels can be created to bring other languages into Polyglot Notebooks. You don’t have to use all of the different IKernelCommandHandler request types that were discussed here for every kernel. You can mix and match based on your specific needs.

Full source code for this project: https://github.com/CodePenguin/codepenguin.github.io/tree/main/assets/2023/08/custom-polyglot-notebook-kernel/PolyglotCalculatorKernel/

If you have questions or comments, feel free to post in the discussion: https://github.com/CodePenguin/codepenguin.github.io/discussions/6

OpenID Connect - Dotnet Desktop Application

2023-05-13T23:00:00+00:00

Recently, I’ve been deep diving into JSON Web Tokens (JWT) and the OpenID Connect protocols. My end goal was to be able to use the OpenID Connect Authorization Flow to add authentication and authorization to a native desktop application using Dotnet.

The basic authentication workflow for a native/desktop/console/ application is as follows (RFC 8252 Section 4.1):

Application retrieves the following from the Authorization Server’s OpenID Connection Configuration Endpoint:
- Signing Keys
- Authorization Endpoint
- Token Endpoint
Application opens the system default browser to the Authorization Endpoint URI including the following details as query string parameters:
- Response Type = “code”
- Client ID
- Redirect URI
- Scope (Must include at least openid)
- State (Secure random value)
- Code Challenge
- Code Challenge Method
Authorization Server’s Authorization Endpoint receives the authorization request and then authenticates and authorizes the user.
Authorization server issues an authorization code and redirects the browser to the specified Redirect URI with the response appended to it as query string parameters.
Application receives the Redirect URI and retrieves the authentication code and validates the state from the response.
Application sends the authentication code to the Token Endpoint including the following details:
- Grant Type = “authorization_code”
- Client ID
- Authentication Code
- Redirect URI
- Code Verifier
Authorization server validates the authorization code and issues the ID and Access Tokens.
Application validates the ID Token using the Signing Keys obtained previously.

There is a lot to unpack in each of these steps but we will cover everything as we implement them.

Configuring the Authentication Server

Since I am mainly focusing on the client-side of things for this round of investigation, Auth0 provided a great starting point for the server-side of things as it fully supports the OpenID Connect Authorization Code Flow. You can quickly sign up for a free account that gives you plenty of capabilities to get started.

Sign up for an Auth0 account.
Create a new Native Application (Mobile, desktop, CLI)
Copy down the Domain and Client ID listed on the Settings tab.
Set the Callback URLs to: http://127.0.0.1:49152/callback
Set the Allowed Web Origins to: http://127.0.0.1:49152
Create two roles: Admin, Guest
Create a test user with the Admin role.

The following values are used for all example outputs below:

Domain: dev-MyDotnetApplication.us.auth0.com
Client ID: OiCPeU0i214x6nJUq4YG4ECUlPOSUAlt

Auth0 User Roles

Auth0 does not add the roles to the ID Token JWT by default. We will use a login action workflow to automatically add the roles as custom namespaced claims to the ID and Access tokens that are generated by Auth0:

exports.onExecutePostLogin = async (event, api) => {
  const namespace = event.client.client_id;

  if (event.authorization) {
    // Set claims 
    api.idToken.setCustomClaim(`${namespace}/roles`, event.authorization.roles);
    api.accessToken.setCustomClaim(`${namespace}/roles`, event.authorization.roles);
  }
};

Note: We are using the Client ID from the authorization request as the namespace because the “roles” claim is restricted by Auth0 and cannot be set by the login actions. If you try, it will be silently ignored and the token will be untouched. If our Client ID was “XYZ123” then a custom claim “XYZ123_roles” will be added for each role. The Authorization:RoleClaimType configuration option must match this value.

Create the Dotnet Console Application

For simplicity we will be using a Dotnet Console Application but the same workflow and implementation can work for User Interface applications as well.

From a command line execute the following to create the console application project:

dotnet new worker -n OpenIdConnectConsoleTest
cd OpenIdConnectConsoleTest
dotnet new class -n AuthenticationSettings
dotnet new class -n AuthorizationCallbackResponse
dotnet new interface -n ICallbackListener
dotnet new class -n CallbackListenerFactory
dotnet new class -n HttpCallbackListener
dotnet add package Microsoft.IdentityModel.Protocols.OpenIdConnect

We will be using the models defined in System.IdentityModel.Tokens.Jwt since Microsoft has already provided the proper DTO classes that are needed to deserialize the requests and validate a JWT. We could create each one on our own but the models are just simple classes with the required properties so we would just be duplicating the effort.

The same thing goes for Microsoft.IdentityModel.Protocols.OpenIdConnect which contains the DTO classes for working with the OpenID Connect protocol.

AuthenticationSettings Class

Edit AuthenticationSettings.cs with the following contents:

namespace OpenIdConnectConsoleTest;

public class AuthenticationSettings
{
    public string Audience { get; set; } = string.Empty;
    public string Authority { get; set; } = string.Empty;
    public string ClientId { get; set; } = string.Empty;
    public string ClientSecret { get; set; } = string.Empty;
    public int RedirectUriPort { get; set; } = 0;
    public string RoleClaimType { get; set; } = "roles";
}

We will use the dotnet user-secrets command to setup the configuration for items we need. Execute the following from a command line from the project folder with the placeholders replace with your Auth0 values from earlier:

dotnet user-secrets clear
dotnet user-secrets set "Authentication:Authority" "https://{Auth0 Domain}/"
dotnet user-secrets set "Authentication:ClientId" "{Auth0 Client ID}"
dotnet user-secrets set "Authentication:RedirectUriPort" "49152"
dotnet user-secrets set "Authentication:RoleClaimType" "{Auth0 Client ID}/roles"

Edit appsettings.Development.json and change the Logging:LogLevel:Default value to Debug. There is lot of debug logging in the example code below so you can view responses. In a real world application you should not log any sensitive information such as tokens.

AuthorizationCallbackResponse Class

Edit AuthorizationCallbackResponse.cs with the following contents:

namespace OpenIdConnectConsoleTest;

public class AuthorizationCallbackResponse
{
    public string AuthorizationCode { get; init; } = string.Empty;
    public string RedirectUrl { get; init; } = string.Empty;
    public string State { get; init; } = string.Empty;
}

This class will be used to return all of the data required to validate an authorization callback.

Worker.cs Class

Edit Worker.cs and add the following using clauses:

using Microsoft.IdentityModel.Protocols;
using Microsoft.IdentityModel.Protocols.OpenIdConnect;
using Microsoft.IdentityModel.Tokens;
using System.Diagnostics;
using System.IdentityModel.Tokens.Jwt;
using System.Runtime.InteropServices;
using System.Security.Claims;
using System.Security.Cryptography;
using System.Text;
using System.Text.Encodings.Web;
using System.Web;

Add the following lines above the ILogger private property:

    private readonly AuthenticationSettings _authSettings;
    private readonly IHostApplicationLifetime _hostApplicationLifetime;
    private readonly static HttpClient _httpClient = new();

Update the constructor:

    public Worker(
        ILogger<Worker> logger,
        IHostApplicationLifetime hostApplicationLifetime,
        IConfiguration configuration)
    {
        _logger = logger;
        _hostApplicationLifetime = hostApplicationLifetime;
        _authSettings = configuration.GetSection("Authentication").Get<AuthenticationSettings>() ?? new AuthenticationSettings();
    }

_authSettings will contain all of the configuration options that we need from the Authentication section. This will come from the values that we stored in user-secrets previously. _hostApplicationLifetime will be used to terminate the console application when required.

ExecuteAsync Method

The Worker class is instantiated on startup as a Hosted BackgroundService. The lifetime of our console application will be based on the WorkerService. Once the ExecuteAsync method finishes executing, the console application will end.

Edit Worker.cs and change ExecuteAsync to the following:

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            ClaimsPrincipal user = await AuthenticateExternalUserAsync(stoppingToken);
            var userId = user.Claims?.FirstOrDefault(c => c.Type == "sub")?.Value;
            var email = user.Claims?.FirstOrDefault(c => c.Type == "email")?.Value;

            _logger.LogInformation("Authenticated as Id: {userId} Name: {name} Email: {email} Claims: {claims}",
                userId,
                user.Identity?.Name,
                email,
                user.Claims);

            _logger.LogInformation("Is in role Admin: {response}", user.IsInRole("Admin"));
            _logger.LogInformation("Is in role Guest: {response}", user.IsInRole("Guest"));

            // Perform other actions here

            _hostApplicationLifetime.StopApplication();
        }
        catch (Exception ex)
        {
            _logger.LogError("Error: {message}", ex.Message);
            Environment.Exit(1);
        }
    }

This will represent the main functionality of the console application. We first authenticate an external user in AuthenticateExternalUserAsync and if authentication was successful we display some user information including User ID, Email, and check to see what roles the user may be a part of. Finally, _hostApplicationLifetime.StopApplication() is called to indicate that the application should exit.

If any exceptions are raised then the application is terminated with a non-zero exit code.

AuthenticateExternalUserAsync Function

    private async Task<ClaimsPrincipal> AuthenticateExternalUserAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Authenticating with {authority}...", _authSettings.Authority);
        OpenIdConnectConfiguration openIdConfiguration = await RetrieveOpenIdConnectConfigurationAsync(stoppingToken);
        string codeVerifier = Base64UrlEncoder.Encode(RandomNumberGenerator.GetBytes(32));
        AuthorizationCallbackResponse response = await RequestAuthorizationCodeAsync(codeVerifier, openIdConfiguration, stoppingToken);
        OpenIdConnectMessage tokenResponse = await RequestTokenAsync(response.RedirectUrl, response.AuthorizationCode, codeVerifier, openIdConfiguration, stoppingToken);
        ClaimsPrincipal claimsPrincipal = ValidateToken(tokenResponse.IdToken, _authSettings.ClientId, openIdConfiguration);
        return claimsPrincipal;
    }

The AuthenticateExternalUserAsync function follows the OpenID Connect authentication workflow. The steps of the workflow are broken down into individual methods.

First, we retrieve the OpenID Connect configuration information in RetrieveOpenIdConnectConfigurationAsync. We generate a random codeVerifier value that will be used to protect our authorization request from interception which we will discuss in more detail in RequestAuthorizationCodeAsync and RequestTokenAsync. For now we just need to know that it is a secure random 32 byte value encoded in URL safe Base64 encoding. We request the authorization code in RequestAuthorizationCodeAsync and then immediately use it to request ID and Access Tokens in RequestTokenAsync. Finally, we call ValidateToken to validate the ID Token and turn it into a Dotnet Identity ClaimsPrincipal that is returned representing the authenticated user.

RetrieveOpenIdConnectConfigurationAsync Function

OpenID Connect uses multiple endpoints that are called to perform the various steps of the protocol. These usually are easy to guess based on the base URI of the authentication server.

For example, if our authentication server base URL was https://dev-MyDotnetApplication.us.auth0.com then you could assume that the Authorization Endpoint would be "https://dev-MyDotnetApplication.us.auth0.com/authorize and the Token Endpoint would be https://dev-MyDotnetApplication.us.auth0.com/oauth/token. But this can be error prone and it is a pain to have to keep track of all of the various endpoint URLs individually. Instead, the OpenID Connect protocol Section 4 provides a well-known URL https://{Base URL}/.well-known/openid-configuration to the authorization server base URL like https://dev-MyDotnetApplication.us.auth0.com/.well-known/openid-configuration. This endpoint returns a JSON object that contains properties for all of the various endpoints and information we will need:

{
    "issuer": "https://dev-MyDotnetApplication.us.auth0.com/",
    "authorization_endpoint": "https://dev-MyDotnetApplication.us.auth0.com/authorize",
    "token_endpoint": "https://dev-MyDotnetApplication.us.auth0.com/oauth/token",
    "jwks_uri": "https://dev-MyDotnetApplication.us.auth0.com/.well-known/jwks.json",
    ...
}

We just need to send an HTTP GET request to the configuration endpoint and then use the URLs in the JSON authorization_endpoint and token_endpoint properties. The JSON jwks_uri property contains a URL used to retrieve the JSON Web Keys that contains the Public Keys needed to verify the tokens that we receive from the token endpoint.

Microsoft has already provided classes that maintain and cache this configuration data for us. The OpenIdConnectConfigurationRetriever and OpenIdConnectConfiguration classes retrieve and store the OpenID Connect configuration data for us. We can then just use the AuthorizationEndpoint, TokenEndpoint and SigningKeys properties of the retrieved OpenIdConnectConfiguration object when we need those values. This makes RetrieveOpenIdConnectConfigurationAsync really trivial:

    private async Task<OpenIdConnectConfiguration> RetrieveOpenIdConnectConfigurationAsync(CancellationToken cancellationToken)
    {
        var uri = _authSettings.Authority.TrimEnd('/') + "/.well-known/openid-configuration";
        _logger.LogInformation("Retrieving configuration from {endpoint}...", uri);
        var configurationManager = new ConfigurationManager<OpenIdConnectConfiguration>(
            uri,
            new OpenIdConnectConfigurationRetriever(),
            new HttpDocumentRetriever());
        return await configurationManager.GetConfigurationAsync(cancellationToken);
    }

RequestAuthorizationCodeAsync Function

This function has a bit more code but isn’t really doing anything complicated. At a high level it does the following:

Starts an ICallbackListener instance and gets the Redirect URI.
Generates an authorization URL including the Redirect URI.
Opens the authorization URL in the default system browser.
Waits for the browser to redirect to the Redirect URI via the listener.
Validates the response and returns the authorization token.

    private async Task<AuthorizationCallbackResponse> RequestAuthorizationCodeAsync(string codeVerifier, OpenIdConnectConfiguration openIdConfiguration, CancellationToken stoppingToken)
    {
        _logger.LogInformation("Requesting authorization code from {endpoint}...", openIdConfiguration.AuthorizationEndpoint);

        // Generate values used to protect and verify the request
        var requestState = Base64UrlEncoder.Encode(RandomNumberGenerator.GetBytes(16));
        var codeChallenge = Base64UrlEncoder.Encode(SHA256.HashData(Encoding.ASCII.GetBytes(codeVerifier)));

        // Start the listener and generate the Redirect URI
        ICallbackListener listener = CallbackListenerFactory.GetCallbackListener();
        listener.Listen(_authSettings, out var redirectUri);

        // Build the authorization request URL
        var parameters = new Dictionary<string, string>
        {
            { "response_type", "code" },
            { "client_id", _authSettings.ClientId },
            { "redirect_uri", redirectUri },
            { "scope", "openid email profile" },
            { "state", requestState },
            { "code_challenge", codeChallenge },
            { "code_challenge_method", "S256" },
        };

        if (!string.IsNullOrEmpty(_authSettings.Audience))
        {
            parameters.Add("audience", _authSettings.Audience);
        }

        string url = BuildUrl(openIdConfiguration.AuthorizationEndpoint, parameters);

        // Open System Browser and wait for the redirect response URI
        _logger.LogInformation("Opening default browser...");
        _logger.LogDebug("Request URL: {request}", url);

        OpenSystemBrowser(url);

        _logger.LogInformation("Waiting for response from browser...");
        var responseUri = await listener.WaitForResponseAsync(stoppingToken);

        _logger.LogDebug("Response URL: {url}", responseUri);

        string? error = null;
        string? errorDescription = null;

        // Verify the redirect response URI exactly matches the Redirect URI sent in the authorization request
        if (!responseUri!.ToString().StartsWith(redirectUri))
        {
            error = "Invalid Response";
            errorDescription = $"The response does not match the redirect URI: {redirectUri}";
        }

        // Check response for errors
        var queryString = HttpUtility.ParseQueryString(responseUri?.Query ?? string.Empty);
        error ??= queryString["error"];
        if (error != null)
        {
            errorDescription ??= queryString["error_description"] ?? "Unknown Error";
            _logger.LogError("Authorization request failed: {error} - {description}", error, errorDescription);
            throw new Exception($"Authorization request failed: {error} - {errorDescription}");
        }

        var callbackResponse = new AuthorizationCallbackResponse
        {
            AuthorizationCode = queryString["code"] ?? string.Empty,
            RedirectUrl = redirectUri,
            State = queryString["state"] ?? string.Empty
        };

        // Verify the state matches what was sent in the authorization request
        if ((requestState != callbackResponse.State) || string.IsNullOrEmpty(callbackResponse.AuthorizationCode))
        {
            throw new Exception("Invalid authorization state");
        }
        return callbackResponse;
    }

When the system browser is done authenticating and authorizing or request, it will redirect the user to the Redirect URI we provided with the response as query string parameters. The redirect response will be handled by the ICallbackListener interface implementation obtained by the CallbackListenerFactory that will be discussed later but for right now all that we need to know is that they handle listening for and retrieving the redirect response URI from the system browser.

One important thing to point out here is that code_challenge and code_challenge_method values are included in the authorization request. This implements the PKCE workflow covered in the RFC 7636 Proof Key for Code Exchange by OAuth Public Clients Section 4.1 and is used to prevent malicious interception and tampering of the data. This value is generated from the Code Verifier mentioned earlier that was created by generating 32 bytes of secure random data and then Base64 URL encoding the bytes. The Code Challenge is then prepared using the S256 Code Challenge Method by taking the Code Verifier and applying the SHA-256 hashing algorithm and then Base64 URL encoding the result again.

An example generated URL is provided below:

https://dev-MyDotnetApplication.us.auth0.com:443/authorize?response_type=code&client_id=OiCPeU0i214x6nJUq4YG4ECUlPOSUAlt&redirect_uri=http%3A%2F%2F127.0.0.1%3A49152%2Fcallback&scope=openid%20email%20profile&state=lZ01ygDKM6JhNC0iwcNLsA&code_challenge=n3KyPPCdoUaY44czXEq_Mb_-k6UqqjscXAV__Z5WjMA&code_challenge_method=S256

When the response is received, the application must verify that the URL used matches the Redirect URI and that the state returned in the response matches the state value sent in the authorization request.

An example response is provided below:

http://127.0.0.1:49152/callback?code=kBVS3EnU_F-mjPTsKaHLWcz6muM4sxAuowBaEmHn9239t&state=lZ01ygDKM6JhNC0iwcNLsA

The value returned in the code query string parameter of the URL is then returned as the authorization code.

BuildUrl Function

This is just a quick helper function that takes a base URL and dictionary of string values and generates a URL with properly encoded query string parameters.

    private static string BuildUrl(string baseUrl, Dictionary<string, string> queryParameters)
    {
        var queryBuilder = new StringBuilder();
        foreach (var pair in queryParameters)
        {
            queryBuilder.Append('&');
            queryBuilder.Append(UrlEncoder.Default.Encode(pair.Key));
            queryBuilder.Append('=');
            queryBuilder.Append(UrlEncoder.Default.Encode(pair.Value));
        }
        var uri = new UriBuilder(baseUrl)
        {
            Query = queryBuilder.ToString().Trim('&')
        };
        return uri.ToString();
    }

OpenSystemBrowser Method

This method just opens the specified URL in the default system browser.

All the actual authentication and authorization steps of this workflow happen in the browser. The system browser allows the user to use normal login workflows such as Password Managers and Multi-factor authentication and protects the users actual credentials from being leaked to the requesting application.

    private void OpenSystemBrowser(string url)
    {
        if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
        {
            Process.Start(new ProcessStartInfo
            {
                FileName = url,
                UseShellExecute = true,
            });
        }
        else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
        {
            Process.Start("xdg-open", url);
        }
        else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
        {
            Process.Start("open", url);
        }
        else
        {
            throw new NotSupportedException("Cannot open default browser");
        }
    }

RequestTokenAsync Function

The authorization code is not very useful on its own. It is only used for immediately requesting ID and Access tokens using the Token Endpoint and an HTTP Post with specific Form URL Encoded parameters.
The authorization_code, redirect_uri, and code_verifier are included in the parameters.
The code_verifier is the Code Verifier value that was originally generated. The Authentication Server will use the same S256 Challenge Method to verify that the Code Verifier given is what created the Code Challenge when the authorization code was requested. This allows the authentication server to validate that the token request came from the same client who originated the authorization code request and that nothing was tampered with during transmission.

An example of the HTTP POST Form URL Encoded contents is provided below:

grant_type=authorization_code&client_id=OiCPeU0i214x6nJUq4YG4ECUlPOSUAlt&code=kBVS3EnU_F-mjPTsKaHLWcz6muM4sxAuowBaEmHn9239t&redirect_uri=http%3A%2F%2F127.0.0.1%3A49152%2Fcallback&code_verifier=YXlDyKO3oD6Fli-NZSEN4CCHvN5OeBNw0ctAEGFBP1g

    private async Task<OpenIdConnectMessage> RequestTokenAsync(string redirectUri, string authorizationCode, string codeVerifier, OpenIdConnectConfiguration openIdConfiguration, CancellationToken stoppingToken)
    {
        _logger.LogInformation("Requesting access token from {endpoint}...", openIdConfiguration.TokenEndpoint);
        var parameters = new Dictionary<string, string>
        {
            { "grant_type", "authorization_code" },
            { "client_id", _authSettings.ClientId },
            { "code", authorizationCode },
            { "redirect_uri", redirectUri },
            { "code_verifier", codeVerifier }
        };

        if (!string.IsNullOrEmpty(_authSettings.ClientSecret))
        {
            parameters.Add("client_secret", _authSettings.ClientSecret);
        }

        var request = new HttpRequestMessage(HttpMethod.Post, openIdConfiguration.TokenEndpoint)
        {
            Content = new FormUrlEncodedContent(parameters)
        };

        _logger.LogDebug("Request: {request}, Form Parameters: {parameters}", request, parameters);
        var response = _httpClient.Send(request, stoppingToken);
        var json = await response.Content.ReadAsStringAsync(stoppingToken);
        _logger.LogDebug("Response: {json}", json);
        if (!response.IsSuccessStatusCode)
        {
            throw new Exception($"Token request failed: {response.ReasonPhrase}");
        }
        return new OpenIdConnectMessage(json);
    }

If everything checks out, the authentication server will then respond with the tokens which are then deserialized to an OpenIdConnectMessage:

{
    "access_token": "eyJ...MWw",
    "id_token": "eyJ...0kg",
    "scope": "openid profile email",
    "expires_in": 86400,
    "token_type": "Bearer"
}

The ID Token is a JWT token that can be decoded using the JWT.IO Debugger. The token’s payload will contain data similar to the following:

{
  "OiCPeU0i214x6nJUq4YG4ECUlPOSUAlt/roles": [
    "Admin"
  ],
  "nickname": "codepenguin",
  "name": "Code Penguin",
  "picture": "https://s.gravatar.com/avatar/7b6...a13?s=480&r=pg&d=https%3A%2F%2Fcdn.auth0.com%2Favatars%2Fco.png",
  "updated_at": "2023-05-11T01:05:37.918Z",
  "email": "email@example.com",
  "email_verified": true,
  "iss": "https://dev-MyDotnetApplication.us.auth0.com/",
  "aud": "OiCPeU0i214x6nJUq4YG4ECUlPOSUAlt",
  "iat": 1684002348,
  "exp": 1684038348,
  "sub": "auth0|640...e06",
  "sid": "qGH...eVbl"
}

ValidateToken Function

This function takes a JWT token and validates it based on the OpenID Connect Configuration.
The Signing Keys are used to validate that the token was produced by the person holding the corresponding private key. We set the RoleClaimType so that the user roles will be taken from the specific claims that our Auth0 Action added to the ID Token.

If the token is valid, a `ClaimsPrincipal

    private ClaimsPrincipal ValidateToken(string token, string audience, OpenIdConnectConfiguration openIdConfiguration)
    {
        _logger.LogInformation("Validating token...");
        JwtSecurityTokenHandler.DefaultInboundClaimTypeMap.Clear();
        var tokenHandler = new JwtSecurityTokenHandler();
        tokenHandler.InboundClaimTypeMap.Clear();
        var validationParameters = new TokenValidationParameters
        {
            ClockSkew = TimeSpan.Zero,
            IssuerSigningKeys = openIdConfiguration.SigningKeys,
            NameClaimType = "name",
            RequireExpirationTime = true,
            RequireSignedTokens = true,
            RoleClaimType = _authSettings.RoleClaimType,
            ValidateAudience = true,
            ValidateIssuerSigningKey = true,
            ValidateIssuer = true,
            ValidateLifetime = true,
            ValidAudience = audience,
            ValidIssuer = _authSettings.Authority,
        };
        ClaimsPrincipal claimsPrincipal = tokenHandler.ValidateToken(token, validationParameters, out var validatedToken);
        _logger.LogDebug("Validated token: {token}", validatedToken);
        return claimsPrincipal;
    }

Retrieving the Authorization Code

The Authorization Code workflow for Native apps (RFC 8252 Section 7) outlines three redirect URI options for retrieving this data:

It doesn’t really matter which way you retrieve the redirect response URI as the actual validation and usage of it is the same and was already covered. The ICallbackListener instances are responsible for retrieving the redirect response URI from the browser so we can retrieve the authorization code. For this article we will focus on the last one “Loopback Interface Redirection” for simplicity but the different ICallbackListener implementations could be created to support the other redirect URI options.

ICallbackListener Interface

Edit ICallbackListener.cs with the following contents:

namespace OpenIdConnectConsoleTest;

public interface ICallbackListener
{
    public void Listen(AuthenticationSettings authSettings, out string redirectUri);
    public Task<Uri?> WaitForResponseAsync(CancellationToken cancellationToken);
}

Since we will need the Redirect URI at other steps in the workflow, the Listen method of ICallbackListener will provide the required Redirect URI for the specific redirect URI option implementation so we can send it to the Authorization Endpoint. The WaitForResponseAsync method will return the redirect response URI that will contain the response values from the authorization request.

CallbackListenerFactory Class

We are only focusing on the “Loopback Interface Redirection” option for this article so the CallbackListenerFactory class just returns a new instance of HttpCallbackListener. In the future we can swap out the implementation for one of the other redirect URI options.

Edit CallbackListenerFactory.cs with the following contents:

namespace OpenIdConnectConsoleTest;

public static class CallbackListenerFactory
{
    public static ICallbackListener GetCallbackListener()
    {
        return new HttpCallbackListener();
    }
}

HttpCallbackListener Class

This class needs to implement the ICallbackListener interface and have a private field to hold the HttpListener instance that will actually be listening for the redirect URI response.

Edit HttpCallbackListener.cs and update the class with the following contents:

using System.Net;
using System.Net.Sockets;

namespace OpenIdConnectConsoleTest;

public class HttpCallbackListener : ICallbackListener
{
    private HttpListener? _listener;

Listen Method

    public void Listen(AuthenticationSettings authSettings, out string redirectUri)
    {
        int redirectUriPort = GetRedirectUriPort(authSettings.RedirectUriPort);

        _listener = new HttpListener();
        _listener.Prefixes.Add($"http://127.0.0.1:{redirectUriPort}/");
        _listener.Start();

        redirectUri = $"http://127.0.0.1:{redirectUriPort}/callback";
    }

This function just returns a local redirect url based with the specified port and configured path and starts the HttpListener on the specified redirect URI.

For our configuration with RedirectUriPort = 49152 and RedirectUriPath = “callback” the resulting Redirect URI would be:

http://127.0.0.1:49152/callback

GetRedirectUriPort Function

    private static int GetRedirectUriPort(int defaultRedirectUriPort)
    {
        // Use a specific port if configured since Auth0 does not support randomly assigned ports
        if (defaultRedirectUriPort != 0)
        {
            return defaultRedirectUriPort;
        }
        // Retrieve an assigned port from the operating system
        var listener = new TcpListener(IPAddress.Loopback, 0);
        listener.Start();
        var port = ((IPEndPoint)listener.LocalEndpoint).Port;
        listener.Stop();
        return port;
    }

This function is generally used to get a randomly assigned port from the operating system that will be used for the Redirect URI listener. Here we just attempt to open a TCP Listener with port 0 so that the operating system will assign us a port. We then close the listener and then use that port for our HTTP Listener. The technique was borrowed from the Google OAuth for Apps: Sample Console Application for Windows.

RFC 8252 Section 7.3 states:

“The authorization server MUST allow any port to be specified at the time of the request for loopback IP redirect URIs, to accommodate clients that obtain an available ephemeral port from the operating system at the time of the request.”

Unfortunately, Auth0 does not currently support randomly assigned ports so you must specify a port in the Authentication settings and it must match the port used when setting up the Auth0 Callback URLs mentioned previously. For other authentication servers that support randomly defined ports, leave the RedirectUriPort setting as 0.

WaitForResponseAsync

    public async Task<Uri?> WaitForResponseAsync(CancellationToken cancellationToken)
    {
        var context = await _listener!.GetContextAsync().WaitAsync(TimeSpan.FromMinutes(5), cancellationToken);
        var request = context.Request;
        using (var writer = new StreamWriter(context.Response.OutputStream))
        {
            writer.WriteLine($"<HTML><BODY>Please return to the application.</BODY></HTML>");
            writer.Flush();
        }
        _listener.Stop();
        return request.Url;
    }

This function waits for the browser to redirect to the specified redirect URI where the HttpListener instance is listening. There is a 5 minute timeout here just so we are not waiting indefinitely.

An HTML response is sent back to the browser so the user knows they should return to the application.

Finally the URL of the request is returned as our redirect response URI.

Testing out the console application

Execute the console application by calling dotnet run.

The system default browser should pop up with a prompt similar to the following:

The browser will then change to show the message “Authorization code received” if everything was successful. Returning to the console application will then show information about the authorized user similar to the following:

info: OpenIdConnectConsoleTest.Worker[0]
      Authenticating with https://dev-MyDotnetApplication.us.auth0.com/...
info: OpenIdConnectConsoleTest.Worker[0]
      Retrieving configuration from https://dev-MyDotnetApplication.us.auth0.com:443/.well-known/openid-configuration...
info: OpenIdConnectConsoleTest.Worker[0]
      Requesting authorization code from https://dev-MyDotnetApplication.us.auth0.com/authorize...
info: OpenIdConnectConsoleTest.Worker[0]
      Starting HTTP Listener...
info: OpenIdConnectConsoleTest.Worker[0]
      Opening default browser...
info: OpenIdConnectConsoleTest.Worker[0]
      Waiting for response from browser...
info: OpenIdConnectConsoleTest.Worker[0]
      Authorization code received
info: OpenIdConnectConsoleTest.Worker[0]
      Requesting access token from https://dev-MyDotnetApplication.us.auth0.com/oauth/token...
info: OpenIdConnectConsoleTest.Worker[0]
      Validating token...
info: OpenIdConnectConsoleTest.Worker[0]
      Authenticated as Id: auth0|640...e06 Name: Code Penguin Email: email@example.com Claims: OiCPeU0i214x6nJUq4YG4ECUlPOSUAlt/roles: Admin, nickname: codepenguin, name: Code Penguin, picture: https://s.gravatar.com/avatar/7b6...a13?s=480&r=pg&d=https%3A%2F%2Fcdn.auth0.com%2Favatars%2Fco.png, updated_at: 2023-05-11T01:05:37.9180000Z, email: email@example.com, email_verified: true, iss: https://dev-MyDotnetApplication.us.auth0.com/, aud: OiCPeU0i214x6nJUq4YG4ECUlPOSUAlt, iat: 1684006121, exp: 1684042121, sub: auth0|640...e06, sid: qGH...Vbl
info: OpenIdConnectConsoleTest.Worker[0]
      Is in role Admin: True
info: OpenIdConnectConsoleTest.Worker[0]
      Is in role Guest: False
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

Microsoft Identity Platform

The code and workflows used here should work with any OpenID Connect compliant identity Provider. The Microsoft Identity Platform can be used as well.

Perform the following steps register a new application and setup the appropriate roles:

Sign up for an Azure account.
Register a new application via Azure Active Directory -> App Registrations.
Set the Redirect URI to: Public client/native (mobile & desktop) and http://127.0.0.1/callback
Copy down the Directory (tenant) ID and {Application (client) ID listed on the Overview screen.
Create two App roles: Admin, Guest
Create a test user.
Use Enterprise Applications -> Users and groups to add the Admin role to the test user.

Execute the following to configure the authentication settings for the Microsoft Identity Platform:

dotnet user-secrets clear
dotnet user-secrets set "Authentication:Authority" "https://login.microsoftonline.com/{Directory (tenant) ID}/v2.0"
dotnet user-secrets set "Authentication:ClientId" "{Application (client) ID}"

The RoleClaimType is not included in the configuration because the Microsoft Identity Platform defaults to using the reserved roles claim which is the default value.

Google’s OAuth 2.0 authentication system

Google’s OAuth 2.0 authentication system can be used as well but does not allow the addition of custom scopes or roles.

Perform the following steps register a new application:

Create a new project at https://console.developers.google.com/.
Configure the OAuth Consent Screen.
Create a new OAuth Client ID (https://console.developers.google.com/apis/credentials and choose Application Type = “Desktop App”
Copy down the Client ID and Client Secret listed on the confirmation screen.

dotnet user-secrets clear
dotnet user-secrets set "Authentication:Authority" "https://accounts.google.com"
dotnet user-secrets set "Authentication:ClientId" "{Client ID}"
dotnet user-secrets set "Authentication:ClientSecret" "{Client Secret}"

An interesting thing to note here is that we are including the Client Secret as well. Generally, for desktop applications using the Authorization Code workflow, you wouldn’t need to include the Client Secret because it cannot be properly secured in the desktop application. In this instance, Google is not treating it as secret as pointed out in their documentation for installed applications:

“The process results in a client ID and, in some cases, a client secret, which you embed in the source code of your application. (In this context, the client secret is obviously not treated as a secret.)”

Conclusion

Using the OpenID Connect Authorization Code Flow we can validate external user credentials in a native desktop application. The system default browser is used so that the user’s credentials are not leaked to the application. If the user is already signed in with their authentication provider, the application is immediately logged in without additional interaction if the user has already authorized the application previously. This greatly increases the speed and security of authenticating a user in native desktop applications.

Full source code for this console application: https://github.com/CodePenguin/codepenguin.github.io/tree/main/assets/2023/05/openid-connect-dotnet-desktop-application/OpenIdConnectConsoleTest

If you have questions or comments, feel free to post in the discussion: https://github.com/CodePenguin/codepenguin.github.io/discussions/5

ShortDash: Cross-platform Shortcut Dashboard

2020-12-04T03:04:55+00:00

A few months ago, I was sitting in a video conference and noticed that the room was getting hot. I have a fan but it was just out of reach and it would have been rude to walk out of the frame in the middle of the meeting. At that point I wished I had one of those digital power plugs that you can control from your phone. I did eventually get one of those plugs but you had to use their dedicated app to control it. I thought it would be much more convenient to have a central dashboard where I could control it. And then it would be nice to be able to execute other things. This led to some tinkering with https://ifttt.com/ and its web request functionality. Soon after, the idea of ShortDash began to form.

ShortDash is a cross-platform shortcut dashboard for your local network. It allows you to create customizable dashboards of shortcuts and actions. You can even turn an old tablet or cell phone into your own personal shortcut dashboard.

A server runs on your local network and allows any device with a web browser to access the dashboards. A cross-platform target application allows actions to be executed on any Windows, Mac or Linux machine. All controlled from a single ShortDash server. Each device can be individually configured to allow full access or just specific dashboards.

Shortcuts can be created by customizing existing actions to execute programs, web requests, etc. Multiple actions can be executed with a single shortcut to create advanced workflows. An open plugin architecture allows anyone to quickly create new actions to fit any workflow required.

With ShortDash, I am now able to use an older generation Kindle Fire tablet as a secondary screen that always has my dashboard display. With one tap on a “Fan On” shortcut on my dashboard, a “Web Request” action is executed on the ShortDash server that sends an HTTPS request to IFTTT which then executes the “Turn plug on” action which finally turns on my fan. When I press the shortcut again, a different message is sent to turn the fan off. IFTTT can integrate with countless other products and services.

That is just one example of what ShortDash can do. The open plugin architecture allows anyone to quickly create new plugins to perform any action that is required. But even if you aren’t a programmer, a lot can be accomplished with the two main core actions: “Web Request” and “Execute Process”. You can quickly create shortcuts that will open your favorite programs or open the websites you read everyday. If you have multiple machines, you can install the Target application and execute actions on Windows, MacOS and Linux, all from the same ShortDash dashboard. The possibilities are endless and we have just barely scratched the surface of the fun and exciting things that you can do with ShortDash.

ShortDash is written in ASP.NET Core using the Blazor Server architecture.

ShortDash is open source and freely available https://github.com/CodePenguin/shortdash/releases for Windows, MacOS and Linux.

Outwitters Sports Network

2013-12-28T23:12:36+00:00

It has been a while since I posted here so I wanted to give some details to what I’ve been working on over the last year. About a year and a half ago I discovered an iOS turn-based strategy game called Outwitters. I was immediately hooked. It had enough variation and strategy to keep my interest. I was playing it all the time and recommending it to everyone who would listen.

After you finish a game you have the ability to re-watch the replays and share them with others. This allowed you to learn new strategies or show off a triumphant victory. Unfortunately, you could only share the replays with someone who had an iOS device and had the Outwitters app installed. I wanted to watch the replays on my desktop so that I could easily compare and contrast different strategies against each other and improve my game.

Outwitters Replay Viewer

I began the process of reverse-engineering the replay protocol (with permission from the original developers) and creating the Outwitters Replay Viewer. I didn’t have any help from the original developers about protocols, file formats, or anything like that so I had to figure that all out on my own based on the data I could extract from the game. It took a bit of time to get everything squared away but in the end it worked out great.

The original version of the replay viewer wasn’t much to look at but it was functional. The second version had very basic graphics to indicate the types of units. But the 3rd version included all the graphics from the original game, looked the best and was the most functional.

iOS on Device Replay

Replay Viewer Version 1

Replay Viewer Version 2

Replay Viewer Version 3

Outwitters Sports Network is born!

Once the replay viewer was completed and people started using it, there began to be a lot of replay data stored on my server and so I started scanning them and creating analytics that allowed you to quickly find specific replays by map, player, unit, etc. This began the transition from just a replay viewer into the Outwitters Sports Network (Abbreviated as OSN). There are statistics, weekly top ranked player lists, leaderboards, popular replays and much more. According to Google Analytics, from August 2012 to December 2013 there have been 86,237 visits, 17,951 unique visitors, and 355,938 page-views. As of December 28, 1013 there have been 21,872 user submitted replays.

OSN Final Replay Viewer

OSN Website

Technical Details

The replay viewer was programmed in Processing and then loaded on the website using Processing.js. Processing didn’t originally support JSON data so I had to use the compiled add-on library and examples at Jer Thorp’s Blog article “Processing, JSON & The New York Times”. Processing.js on the other hand does not support any java add-on libraries like Processing does so I ended up creating a special JavaScript based add-on that would supply Processing.js a library with the same footprint as the original java JSON library. I found a wonderful reference guide on Processing.js’s website called Pomax’s Guide to Processing.js that was invaluable in getting Processing.js and JavaScript to work together. This allowed Processing and Processing.js to use the exact same Processing based code so I didn’t have to support two separate code bases.

The Outwitters Sports Network Website utilizes the following technologies:

PHP and CodeIgniter for the backend processing and website
Bootstrap for the majority of styles on the website
JQuery for the powerful dynamic web interface
Handlebars for client-side templating
Highcharts for client-side charts on the statistics page
MySQL for data storage of statistics and searchable data
JSON text files for storage of full replay data files

Summary

It was definitely a long ride but a pretty enjoyable one. The final product ended up being way different than I had originally planned but I think it came out great. Especially, since it was originally just a proof of concept to see if I could do it. The Outwitters Sports Network is still going strong and still gets quite a few visitors. It is currently in maintenance mode as I’ve pretty much added all the features that I really wanted to be there. I still update the top player ranks every week and keep things running smoothly.

Slipix Version 1.0 for iOS

2011-09-01T23:21:43+00:00

Slipix has been discontinued and will not be updated anymore.

Introduction

Slipix is a unique new puzzle game where you touch and drag different layers of an image until the real picture is revealed. These are not jigsaw puzzles. Each layer is a different design or pattern that create unique challenges.

There are 40 unique levels to master with four difficulty levels. Quick Play mode can generate an endless amount of levels to satisfy any puzzle solver. There is even a head to head multiplayer mode (iPad only) so you can play with your friends.

More levels are on the way!

Click here to check it out in the App Store!

Video

http://www.youtube.com/watch?v=UflM0cVg3wY

Screen Shots

WebSolver: An Interactive Cryptogram Solver for the Web!

2010-06-20T02:48:37+00:00

A great carpenter does not need to know how to make a hammer or saw in order to create a masterpiece.

A while back I wrote a bunch of articles describing how to create your own interactive cryptogram solver using python. I’ve used it a lot in my own personal solving and have had a few friends use it too but it definitely is not a tool for everyone. There are a lot of great tools out there for solving ciphers but many of them are very technical, require a special environment to run, or require programming knowledge and experience. Now, I don’t want to down play the usefulness of these great tools or tell anyone that they shouldn’t learn computer programming (Note: I think all computer users should know the very basics of programming logic as it is so useful for many things). That being said, there are many people out there who love a good challenge but either don’t have the desire or the time to learn to program. Sometimes the greater skill is knowing how to use the tools properly and efficiently. A great carpenter does not need to know how to make a hammer or saw in order to create a masterpiece.

So for all you great carpenters (solvers) out there, I have ported my interactive cryptogram solver to the web! No downloads, no special environments (other than a modern web browser), and no programming skills needed. Just a clean area where you can do what you do best: solving.

Web Solver is available now at http://tools.codepenguin.com/websolver/.

At the moment, the Web Solver only has support for Aristocrats and Patristocrats. I will be adding a lot more as time progresses. I’m still learning so I’ll add my techniques as I learn them. If you have any suggestions or different techniques that could be added, please leave a comment or contact me. I’m always open for suggestions. If you are familiar with my previous interactive solver, then the interface will be very similar. This time there is an integrated help menu available on the right side of the screen that explains all the commands. The interface is mostly keyboard driven as changes can be made more quickly with the keyboard than the mouse. This allows many different types of ciphers to be solved in a similar interface with commands that are specially designed for them.

Quick Overview

Lets run through a quick session of using the Web Solver. I’m going to grab the A-2 cipher from the sample issue of the Cryptogram. When you first open the Web Solver for the first time, it should automatically show you the “Load” Screen. This screen can be accessed at any time by clicking the load button on the right hand side of the screen. The help button is also in that same area providing a handy command reference and usage guide. I’m going to paste/type in the A-2 cipher text into the text area on the loading screen and then click the “Start Solving!” button.

Note: As we go through a few examples of commands, they will be displayed in little terminal boxes in this article for display and usage purposes. Images would be too small and you wouldn’t be able to copy and paste from them. So just be aware that the look will be simulated but pretty close to what the real Web Solver looks like.

For all the cipher types there are three main commands: clear, display, and pattern. Clear just clears the screen in case you want to remove the clutter. Display (shortcut: d) displays the current state of the cipher. This generally will display both the ciphertext and the current state of the plaintext. Since all cipher types are different, the display will vary depending on the type. For example: Aristocrats will have word spacings while Patristocrats will be broken into five letter groups. Lets check out what the display command looks like for our current con.

Aristocrat Solver loaded.
> d
OB ISZDPH *GQG EFBE KZE NZUZPJ SQQO ZE EQ EOFNN AKFA BQT YFP'A EKQTA
-- ------ *--- ---- --- ------ ---- -- -- ----- ---- --- ---'- -----

FA AKD YFA VZAKQTA JDAAZPJ F OQTAKITN QI KFZS.
-- --- --- ------- ------- - -------- -- ----.
>

Useful commands

Now that we can see our ciphertext, lets take a look some patterns. The patterns command (shortcut: p) can be used to quickly look up a pattern and return any words that match it. This command is pretty powerful so definitely read the help on it for more information. Lets just look at a quick example. We’ll take at a few examples from the con:

> p ISZDPH
389 matches found.
absurd acting action admire adults advice advise agency agents aliens almost amount
angels answer anyhow around asking author awhile backed backup barely basket beacon
beauty begins behalf behind beings belong betray beyond bishop blocks blonde bodies
boring bother bought brains breaks breast breath bridge bright brings broken budget
burden buried burned busted buster buying campus candle caring carpet casino castle
caught caused chairs change chapel charge chosen cipher claims client closed closer
closet column comedy coming counts county couple course cousin covers coward credit
crimes cruise crying custom dancer danger dating deacon demons denial deputy design
detail direct double doubts
290 words not shown.

> p OQTAKITN
5 matches found.
adoption handling partners patients ultimate

> p JDAAZPJ
1 match found.
getting

Notice that you can grab the words directly from the ciphertext. You can also search for number based patterns and use filters to narrow down the results. There are four different word lists currently available with varying amounts of words. You can switch to a different word list at any time. We’ll use the set (shortcut: t) command to set the plaintext letters for one of the pattern words that we just found.

> s JDAAZPJ getting
OB ISZDPH *GQG EFBE KZE NZUZPJ SQQO ZE EQ EOFNN AKFA BQT YFP'A EKQTA
-- --ien- *--- ---- -i- -i-ing ---- i- -- ----- t--t --- --n't ----t

FA AKD YFA VZAKQTA JDAAZPJ F OQTAKITN QI KFZS.
-t t-e --t -it---t getting - ---t---- -- --i-.

The set command will except full words or single letters. You can even reset a letter by only giving a ciphertext letter. (Example: s X). Now that we have some letters we can narrow down one of our previous pattern searches that had 389 matches.

> p ISZDPH --ien-
3 matches found.
aliens client friend

That looks much better. These filters can be used to quickly narrow down the results. You can even use wild cards if you don’t know exact lengths.

If you want to see the frequency chart, you can use the frequency command (shortcut: f). It can even give you digraphs and trigraphs if you give it a length.

> f
A:12 Q:9 Z:8 F:8 E:7 K:7 T:5 O:4 P:4 N:4 B:3 I:3 S:3 D:3 J:3 G:2 Y:2 H:1 U:1 V:1

> f 2
AK:4 QT:4 FA:3 TA:3 ZE:2 ZP:2 PJ:2 KF:2 YF:2 KQ:2

> f 3
QTA:3 ZPJ:2 KQT:2

These are only a few of the commands that are available. The help system will give you details for everything with examples. If you have any ideas for new commands, just let me know. They can be easily added and then everyone can have access to new solving techniques.

Save and Continue

Ever been solving a cipher and had to go do something else? Maybe your computer crashed? The Web Solver will automatically save your progress and allow you to come right back where you were as long as you’re using the same computer. All the data is stored locally on your computer.

Command History

Web Solver will keep the history of the last 25 commands that you have used. You can go back through them by pressing up and down. This way you can save some typing if you want to go back to a previously used command.

Conclusion

Web Solver is a solving tool. It won’t do all the work for you but hopefully it will make things easier. It is available now at http://tools.codepenguin.com/websolver/. Please, let me know what you think. Leave a comment or contact me directly. I love to hear your ideas and suggestions. I will continue to improve Web Solver and add new features as time goes on.

Technical Details and Links

For those interested, I’d like to mention the different technologies and techniques that made this possible.

jQuery - A powerful javascript library that makes cross-platform web development a whole lot easier. This is pretty much the core of the entire implementation.
Taffy DB - A JavaScript library that acts as an in memory data layer. This powers the CP List which is a small embedded word list. All the other word lists require an external database connection. CP List can be used without an internet connection.
LazyLoad - A JavaScript library used to load files dynamically only when needed. This is used to load the word list functionality on the fly without taking up a lot of memory.
PersistJS - A JavaScript library that provides client-side persistent storage. This handles all the storage needs and works great in all modern browsers.
How To Create A Sexy Vertical Sliding Panel Using jQuery And CSS3 - This tutorial helped me create the sliding Load and Help panels.

Pattern Words: A Solver’s Staple

2010-06-12T02:08:49+00:00

While working on the Beginner’s Guide to the ACA, there was one technique that consistently showed up in all the research material: Pattern Words! At first I didn’t really understand how great they were. For those who don’t know about them or need a refresher, I’ll discuss a few of the different pattern types that I have come across. Afterwards, I’ll discuss my own Pattern Word Search Tool. Lets get started!

Pattern Types

Fixed Length Patterns

Pattern words have one or more letters that are repeated. These repeated letters provide a clue to the word’s identity. Start by taking a word and assigning numbers to each letter starting at the left with 1 . Use the same number for repeated letters. Now you can look for the number based pattern in a pattern word list to see which words match the pattern. I call these the “Fixed Length Patterns”.

Common pattern words worth memorizing:

1231    123142    12342
that    people    which

122     12343     123314
all     there     little
off     these
see     where
too

Floating Length Patterns

While researching pattern words I came across a few other pattern types that are useful also. Codasaurus.com mentioned a shorted version of the above numeric pattern that only shows the repeated characters. I call these the “Floating Length Patterns”. These are great when you don’t know the full length of the word. First calculate the pattern as above. Then, starting at the front, write down all the positions of the first duplicated letter. Continue this process for all other duplicated letters. This form can be useful when you don’t know the exact length of a word or when multiple word endings are possible. The above examples would be as follows using this method:

14      1426      25
that    people    which

23      35        1534
all     there     little
off     these
see     where
too

High/Low Frequency Patterns

In the comments on my article Social Cryptogram Solving, BOATTAIL mentioned patterns that looked at the frequency of letters. Using a frequency table you can determine which letters are high and low frequency letters. Then replace the letters with H for high and L for low. For English, the high frequency letters are ETAONIRSH. The examples from above would look as follows:

HHHH    LHHLLH    LHHLH
that    people    which

HLL     HHHHH     LHHHLH
all     there     little
off     these
see     where
too

Searching for Pattern Words

You can find pattern word lists on the internet by just searching for them. Most of the time they are a little hard to handle for beginners because they usually come in raw text files and they generally just contain the fixed length patterns. I’ve created a special website that will allow you to search for any of the patterns discussed above in a few different word lists. At the moment I have three different word lists that are searchable. They each have their own strengths and weaknesses. You can access it at http://tools.codepenguin.com/patterns/. The page has instructions for how to use each of the different patterns above and how to apply word filters to weed out the possibilities.

Depending on the word list used, pattern words become less useful simply because they return too many words. That is where the word filters come into play. With word filters, you can shrink the matches for the patterns down to ones that fit your needs. For example: if you look up the pattern 1426 (for people) in the ENABLE2K word list, it returns 130 words. If you know that the first and fourth letters are “p” you can use the filter “p..p*” the matches drop down to 24 words. 24 is much more manageable and realistic.

If you find my pattern search tool useful, please let me know. I’m always open for suggestions on how to improve my tools and for new ideas.

Conclusion

All the different kinds of patterns give you a lot of techniques to aid in solving Aristocrats and Patristocrats. You can get quite far by using them. Pattern word lists are available on the internet and there are some published books that have them too. They are an essential tool for any solver’s tool box.

Beginner’s Guide to the ACA

2010-06-06T23:32:09+00:00

When I first joined the ACA and opened my starter packet, I was eager to jump right in and start solving Aristocrats and Patristocrats. I tried to solve as many as I could. Once I got stuck I would try to learn a different way to solve in hopes that it would get me unstuck. There is so much great information available on solving different cipher types and trying out new techniques. Unfortunately, since there is so much great information, it is difficult to figure out where to start or what resources to use. I found myself frequently getting lost in material too advanced at the early stage and becoming quite frustrated when the information did not penetrate my thick skull.

After one frustrating week, I contacted LIONEL, the ACA Recruitment Committee chairman, in hopes of finding some form of beginner’s guide to help me stay focused. He mentioned the “Young Tyros Newsletter”. It is a wonderful resource and I really recommend all new members and even non-members to read them. They have lots of great insights and tips to get you unstuck. In our discussions, I proposed that we should create a “Beginner’s Guide to the ACA”. I definitely didn’t want to re-hash all the great information out there. My goal was to compile a small list of the most important topics for a beginner to know and then point them in the right direction of all the other great materials out there.

LIONEL put me in contact with a few members who graciously agreed to send me some of their notes on how they solved some of the lower level Aristocrat and Patristocrat cons. Their notes and insights have been a great help in increasing my knowledge and solving abilities. It took quite a long time and many iterations before I was satisfied with it but I finally finished the first version of the “Beginner’s Guide to the ACA!” This would not have been possible without the help of BECASSE, GGMA, LIONEL, MICROPOD, MATANZA, and ZANAC. I am very grateful for their help.

You can download the Beginner’s Guide to the ACA below. Hopefully, it’ll help someone not get frustrated and lose interest in cryptology before they gave it a good chance.

Click here to download!

Creating an interactive cryptogram solver (Part IV)

2009-10-31T23:40:25+00:00

I’ve been working hard on this part of the series because I really wanted the interactive cryptogram solver to make it quick and painless to jump right into solving and still give you plenty of room to expand the functionality and reflect your own style of solving. In this part of the series, we will create solver.py which will become our gateway to solving. It will allow us to quickly select a cipher class that we want to work with. We will also add a self documenting system that will allow us to use the solver without memorizing all the commands or shortcuts that each solver class may use. So, lets just jump right back into the code!

Clearing the screen

While we are using the solver, it is a common place for the screen to get a little crowded. So lets add a new function to our shared.py that will take care of clearing the screen. Clearing the screen is a little complicated because each operating system has it’s own way of handling it. Our clear_screen function will handle all the major players: Windows, Mac, and Linux.

Contents of shared.py (Place at the bottom of the file)

def clear_screen():
  import os
  if os.name == "posix":using the
      # Unix/Linux/MacOS/BSD/etc
      os.system('clear')
  elif os.name in ("nt", "dos", "ce"):
      # DOS/Windows
      os.system('CLS')

All we are really doing here is importing the os and using the os.name to tell which operating system we are in and then executing the appropriate command for clearing the screen.

The Solving Gateway: solver.py

I call solver.py the gateway because this is really the file that you will be executing the most. Everytime we create a new class based on CipherSolver, we’ll want to import it into solver.py so we can access it. I’ll show you how to do that in just a minute. First, lets just show the entire contents of solver.py and then we’ll go through the good parts.

Contents of solver.py

import shared
from cipher import CipherSolver
from aristocrat import AristocratSolver
from patristocrat import PatristocratSolver

if __name__ == '__main__':
  solvers = {}
  for name, obj in globals().items():
    if name.endswith("Solver") and issubclass(obj, CipherSolver):
      solvers[name[:-6]] = obj

  while True:
    shared.clear_screen()
    print "-" * 37
    print "    Interactive Cryptogram Solver"
    print "-" * 37
    print "Type a cipher type or partial (q to quit) [all]: ",
    cmd = raw_input().lower()

    if cmd in ["q","quit"]:
      break
    else:
      list = []
      for name, obj in solvers.items():
        if name.lower().startswith(cmd):
          list.append(name)
      if len(list) > 0:
        list.sort()
        for index in range(1,len(list) + 1):
          print "  {0}. {1}".format(index, list[index-1])
        print "Type number of selection or return to go back: ",
        cmd = raw_input()
        if cmd.isdigit() and (0 < int(cmd) <= len(list)):
          name = list[int(cmd)-1]
          print "nLoaded {0} Solver.".format(name)
          solvers[name]().solve()
      print ""

The first few lines are essentially the most important part of this file. They import all the CipherSolver classes that we want available. You only need to import the your Solver classes, not the Cipher classes.

from cipher import CipherSolver
from aristocrat import AristocratSolver
from patristocrat import PatristocratSolver

Right now we only have three classes to import. We have to import the CipherSolver class because everything is based on that one. This is where you would include any other Solver classes that you create. The pattern is:

from FILENAME import SOLVERCLASS

Just add any new ones below the PatristocratSolver line.

if __name__ == '__main__':
  solvers = {}
  for name, obj in globals().items():
    if name.endswith("Solver") and issubclass(obj, CipherSolver):
      solvers[name[:-6]] = obj

Line 6 is an interesting line of code that you may have seen in other Python files. Python files can be executed on their own or imported from other Python files as we have seen above. One problem that arises is that python tries to execute all lines that are in the file as it is imported. But what if we want to execute code only when our file is executed on its own? That is exactly what this line of code does. It essentially says “If this file is being executed on its own and not imported, then do the following:”. Anything indented below the “if statement” will only be executed when we type:

python solver.py

The rest of the lines in this section walks through all the imported classes and functions in the globals dictionary. Whenever we use the import command, the things we import get stored in the globals dictionary. So here we are scanning globals and pulling out any classes that are based off of our CipherSolver class. We’ll store all the classes and their names so we can use them later.

Note: I’m going to warn you right now, this part of the series contains a lot of code that is purely just for display purposes. We’ll be formatting a lot of things things to display on the screen. I won’t go over every line in these cases. Just know that if you see “print” or “output” then those are things that will be displayed or eventually displayed.

    cmd = raw_input().lower()

    if cmd in ["q","quit"]:
      break
    else:
      list = []
      for name, obj in solvers.items():
        if name.lower().startswith(cmd):
          list.append(name)
      if len(list) > 0:
        list.sort()
        for index in range(1,len(list) + 1):
          print "  {0}. {1}".format(index, list[index-1])
        print "Type number of selection or return to go back: ",
        cmd = raw_input()
        if cmd.isdigit() and (0 < int(cmd) <= len(list)):
          name = list[int(cmd)-1]
          print "nLoaded {0} Solver.".format(name)
          solvers[name]().solve()
      print ""

Here we prompt the user to type in the name or part of a name of the class that they would like to use. We then walk through our list of Solver classes and display any that match the inputted name and display them on the screen. The user then gets to type the number that corresponds with the class on the screen. You can also just press enter to get a list of all available classes. Lets see what the looks like when we run it.

python solver.py
-------------------------------------
    Interactive Cryptogram Solver
-------------------------------------
Type a cipher type or partial (q to quit) [all]:

Lets just press enter here to see all the classes that are loaded:

Aristocrat
Cipher
Patristocrat
Type number of selection or return to go back:

If we type “3” to select Patristocrat and then press enter we get the following:

Type number of selection or return to go back:  3

Loaded Patristocrat Solver.
>

Now we are ready to solve a Patristocrat! Our PatristocratSolver class is automatically loaded and executed. If we want to go back to the selection screen we can just press Ctrl+C at any time.

The Documentation System

I don’t know about you but I have a major problem remembering things sometimes. I even have a T-Shirt that says “Insufficient Memory” on it. So if we create a new solver class for the over sixty different cipher types used in the Cryptogram, there is no way I could remember all the different commands and functions that we would need. It would be very beneficial for us to have documentation on all the different classes we use and to be able to display it whenever we need to. I’m not the greatest at writing documentation especially when the code can change so quickly. So our documentation system would have to be easy to update and not super complicated. Alright, here we go. We are going to put the documentation into the CipherSolver class so most of our changes are going to be in the cipher.py file.

Contents of cipher.py (Add at the top on line 4)

import inspect,types

Here we are importing two modules. The inspect module is going to be the core of our documentation. The python documentation describes this module as:

inspect - Get useful information from live Python objects.

This module can give you all sorts of information about our classes and objects. We’ll see how its used in a minute.

class CipherSolver(object):
  def __init__(self):
    self.cipher = Cipher()
    self.prompt = ">"
    self.maxlinelen = 70
    self.shortcuts = {"?":"_display_help","d":"display"}

The above code is just slightly different from Part III. We are now telling the CipherSolver that it should inherit from the object class. This is the new style of classes in python and just makes sure our code will work in future version of python and that all the special python modules that use objects will work correctly. We also add a new shortcut on line 35. This shortcut lets us type “?” at any time to activate our help system by calling the _display_help function.

One thing to note here, any function that starts with a “_” will be hidden from the documentation system. They won’t be displayed. This allows you to add helper functions that are not available for calling directly. Lets add a few of these little helper functions. (I’m going to go a little out of order here. If you want to see the cipher.py file in its correct order, view the source at the end of this article.)

  def _inherit_docs(self, BaseClass):
    members = inspect.getmembers(self, inspect.ismethod)
    for name, method in members:
      if method.__doc__ == None and hasattr(BaseClass, name):
        method.im_func.__doc__ = getattr(BaseClass, name).__doc__

The _inherit_docs function will walk through all the different overridden functions in our classes and make sure they have their own documentation or use the documentation from the base class. This may sounds confusing at first but we’ll show actual code examples later. Simply put, since all our classes are going to have a display function that just does things a little different for each class, why should we have to repeat the documentation each time? In our documentation system, the PatristocratSolver will look at the AristocratSolver for its documentation and the AristocratSolver will look at CipherSolver for its documentation. Lets add those calls right now. Add the following to the aristocrat.py and patristocrat.py.

Contents of aristocrat.py (Add to the AristocratSolver.init method)

self._inherit_docs(CipherSolver)

Contents of patristocrat.py (Add to the PatristocratSolver.init method)

self._inherit_docs(AristocratSolver)

We need to make one major modification to the CipherSolver.solve method. In the previous parts of this series, we made the solve method handle any methods that are part of the solver class and the cipher class. Our modification is going to change this to only go through the solver class. This might sound like a disadvantage at first, but it will make our solver much more powerful in the end. Change your solve method to the following:

  def solve(self):
    while True:
      try:
        raw_line = raw_input(self.prompt + " ")
        line = re.findall('\'[^\']*\'|\"[^\"]*\"|S+', raw_line)
        line = [item[1:-1] if item[0] in '\'"' else item for item in line]
        print ""
        cmd = line.pop(0)
        attr = None
        if cmd in self.shortcuts:
          cmd = self.shortcuts[cmd]
        if cmd.endswith("?"):
          print self._display_help(cmd[:-1])
        else:
          if hasattr(self, cmd) and cmd != "solve":
            attr = getattr(self, cmd)
          if attr == None:
            print "Unknown command: ", cmd
          else:
            retvalue = None
            if type(attr) == types.MethodType:
                retvalue = attr(*line)
            else:
              if len(line) == 0:
                retvalue = attr
              else:
                setattr(self, cmd, *line)
            if retvalue != None:
              print retvalue
          print ""
      except KeyboardInterrupt:
        print "Exiting Solver"
        break
      except Exception, e:
        print "Error: ", e

The big change here is that we first check to see if the command ends with a question mark. If so, it will pass the command, minus the question mark, to the _display_help function that we’ll discuss in a minute. We also add the support to get and set a properties that are defined in the class.

Now comes the entire documentation system. Its all bundled in a special function called _display_help. Now be prepared, this is a long function.

  def _display_help(self, cmd=None):
    if cmd in self.shortcuts:
      cmd = self.shortcuts[cmd]

    def _get_function_def(name,function):
      specs = inspect.getargspec(function)
      defaults = None
      if specs.defaults:
        defaults = dict(zip(specs.args[-len(specs.defaults):],specs.defaults))
      output = name
      for arg in specs.args:
        if arg != 'self':
          output += " " + arg
          if defaults and arg in defaults:.  I
            output += "=" + repr(defaults[arg])
      return output.  I

    def _display_doc(obj):
      doc = ""
      if inspect.getdoc(obj):
        for line in inspect.getdoc(obj).split("n"):
          doc += "   " + line + "n"
        doc += "n"
      return doc

    cmd_shortcuts = dict([(v,k) for k,v in self.shor.  Itcuts.items()])
    obj = None
    if cmd == None:
      obj = self.__class__
    elif hasattr(self.__class__,cmd):
      obj = getattr(self.__class__,cmd)
    output = ""
    if inspect.isclass(obj):
      output = obj.__name__ + ":nn"
      output += _display_doc(obj)
      member_filter = lambda obj: inspect.ismethod(obj) or isinstance(obj,property)
      members = dict(inspect.getmembers(obj, member_filter)).keys()
      members = [name for name in members if not name.startswith("_") and name != "solve"]
      members.sort()
      output += "  Available Commands:n"
      for index in range(len(members)):
        member = members[index]
        if cmd_shortcuts.get(member):
          member += " [" + cmd_shortcuts.get(member) + "]"
        output += member.rjust(20)
        if (index + 1) % 3 == 0:
          output += "n"
    elif inspect.ismethod(obj) and cmd != "solve":
      output = cmd
      shortcut = dict([(v,k) for k,v in self.shortcuts.items()])
      if shortcut.get(cmd):
        output += " [" + shortcut.get(cmd) + "]"
      output +=":nn" + _display_doc(obj)
      output += "   Usage:n      " + _get_function_def(cmd, obj)
    elif isinstance(obj,property):
      output = cmd
      if obj.fset == None:
        output += " (Read Only)"
      if cmd_shortcuts.get(cmd):
        output += " [" + cmd_shortcuts.get(cmd) + "]"
      output += ":nn" + _display_doc(obj)
      output += "   To display the value:n      " + cmd + "nn"
      if obj.fset != None:
        output += "   To change the value:n      " + _get_function_def(cmd, obj.fset)
    else:
      output = "Unknown command: " + cmd
    return output.rstrip("n") + "n"

It looks daunting at first but is actually quite simple. If you remove all the display code (all the output statements) this is what you’re left with: the function takes a command that you want to see the documentation for and displays it or if you pass in blank, it’ll display all the commands that are available and any shortcuts that are defined. Ok, lets see how this works right now! Fire up sandbox.py for now:

python sandbox.py
> ?

PatristocratSolver:

  Available Commands:
         display [d]  frequency_list [f]            keys [k]
        print_counts             set [s]

>

So if we type “?” It’ll show us all the commands! Notice that the shortcuts are listed in brackets next to the commands. This way you can see them if you have forgotten what they are. Now lets get back to the code. Lets look at a few of the important parts of the _display_help function.

Did you notice that the print_counts function was listed in our command list from above? That isn’t really a command that should be executed in the solver. It’s one of those special helper functions. Lets change its name to _print_counts so that it will be hidden (Note: You will need to change where it is used in aristocrat.py and patristocrat.py too or it will give you an error.).

Lets chop up the _display_help function into the three types of objects that the command passed in can represent. First, a solver class like AristocratSolver. Second, a method of the class like display. Third, a property of the class.

Solver Classes

As we saw before, if no command is passed into the function, then a summary of all available commands is displayed. This will include any shortcuts and properties that are defined for the class. This will also include any commands that are inherited from other classes. Lets look at the code that makes this possible:

      member_filter = lambda obj: inspect.ismethod(obj) or isinstance(obj,property)
      members = dict(inspect.getmembers(obj, member_filter)).keys()

First we create a quick little lambda function that returns True or false. We want to get a list of all methods and properties of our class, so the lambda function will return True only if the “obj” argument is a method or property. Next, we execute the inspect.getmembers function and pass in the obj (which is our solver class at this moment) and our lambda function from above. In python, the inspect.getmembers function would return the following:

>>> import inspect
>>> from patristocrat import PatristocratSolver
>>> member_filter = lambda obj: inspect.ismethod(obj) or isinstance(obj,property)
>>> inspect.getmembers(PatristocratSolver,member_filter)
[('__init__', ),
('_display_help', ),
('_inherit_docs', ),
('_print_counts', ),
('display', ),
('frequency_list', ),
('keys', ),
('set', ),
('solve', ),
('text', )]
>>>

It returns a big list of name/value pairs for all the members that the member_filter evaluated to True. It would be a much longer list if we had not passed in a filter function. The documentation system will use this list to display a nicely formatted display of commands that we can execute.

Methods

When a method name is passed into the function, the doc string for the function is displayed, followed by the appropriate way to call the function and what parameters to pass in. Doc strings are an important part of Python classes and functions. They help provide documentation automatically. Doc strings are placed on the line directly below the declaration of the method. They are started with three quotes “”” and ended with three more quotes “”” Lets add a doc string to the display method and then see how our documentation system will react to it.

  def display(self):
    """Displays the current data on the screen."""
    print self.cipher.text

Here we have add just one line right below the declaration of the display method. The doc string can contain any information you want displayed. It can really be as long as you want and can span multiple lines. Now lets see how it works. Load up the solver and select Patristocrat.

Loaded Patristocrat Solver.
> display?

display [d]:

   Displays the current data on the screen.

   Usage:
      display

>

Thee display isn’t as impressive as if we looked at the frequency_list function. Lets check that one out to better see what is displayed (Note: We haven’t put in a doc string for the frequency_list function yet so you won’t see that):

> frequency_list?

frequency_list [f]:

   Usage:
      frequency_list length=1 text=''

>

We can see the names and default values for the frequency_list. If we don’t pass in anything (by just typing frequency_list and hitting enter), it will automatically pass in a length of 1 and use the internal text value. But we could pass in other lengths if we want to. You don’t have to do anything special to get this to display nicely like this. The code for it is actually quite simple thanks to the inspect module. The inspect code is in the _get_function_def method that we declared above.

      output += "   Usage:n      " + _get_function_def(cmd, obj)

    def _get_function_def(name,function):
      specs = inspect.getargspec(function)
      defaults = None
      if specs.defaults:
        defaults = dict(zip(specs.args[-len(specs.defaults):],specs.defaults))
      output = name
      for arg in specs.args:
        if arg != 'self':
          output += " " + arg
          if defaults and arg in defaults:
            output += "=" + repr(defaults[arg])
      return output

The inspect.getargspec function takes a function as the argument and returns all the information about the function:

>>> specs = inspect.getargspec(PatristocratSolver.frequency_list)
>>> specs
ArgSpec(args=['self', 'length', 'text'], varargs=None, keywords=None, defaults=(1, ''))
>>>

The returned value is called a named tuple. It has all the little chunks of information that we can use. One of the cool things we can do is use the zip function to attach the argument names back to their default values:

>>> defaults = dict(zip(specs.args[-len(specs.defaults):],specs.defaults))
>>> defaults
{'text': '', 'length': 1}
>>>

The rest of the code is just the fluffy display code that makes everything look nice.

Properties

Properties are a great way to get and set different values that our solver will use. We want to be able to get and set the Cipher.text from within our solver class. Lets declare our new property in CipherSolver:

Contents of cipher.py (Add to the CipherSolver class)

  @property
  def text(self):
    """The raw text value used for encoding and decoding."""
    return self.cipher.text

  @text.setter
  def text(self,value):
    self.cipher.text = value

The “@property” tells python that we want the method that follows it to be a property. So now whenever we use the “text” command, the cipher.text will be display. The “@text.setter” tells python that if we try to set the text property, it should change the cipher.text. We can create as many of these kinds of properties we want whenever we want to get and set special values. We’ll probably use them a lot when working with out cipher types.

The display for properties is different from the classes and methods from above. Lets see what our new text property looks like in the documentation system and the solver. Lets load up sandbox.py:

> text?

text:

   The raw text value used for encoding and decoding.

   To display the value:
      text

   To change the value:
      text value

>

So it shows the doc string for the property, how to display it, and also had to change it. Lets display and set our property:

> text

GSRHRHZGVH GLUGSVVNVITVMXBYIL ZWXZHGHBHGVN

> text "THIS IS A TEST"

> text

THIS IS A TEST

>

Now we can easily set the text property whenever we want to start a new con. These properties are really easy to create and are very powerful. Python makes it very easy to enhance our solver with whatever we need to get the job done.

Document Everything!

At this point the implementation of the documentation system is complete. All we have to do now is just add doc strings to all our methods and properties in each of the solver classes.

CipherSolver (cipher.py)

We’ve already added the doc strings to the existing methods and properties in our CipherSolver but lets add one more method that will come in handy. The clear method just clears the screen whenever called. Sometimes when you’d made a lot of changes and there is tons of data on the screen, it is refreshing just to clear the screen before you continue. Add the following code above the display method:

  def clear(self):
    """Clears the screen"""
    shared.clear_screen()

AristocratSolver (aristocrat.py)

Since the AristocratSolver class inherits from CipherSolver, we want to make sure that all our overridden methods like display get their doc strings set correctly so we don’t have to repeat ourselves. We do this by just adding a call to the inherit_docs_ method in the AristocratSolver.__init

class AristocratSolver(CipherSolver):
  def __init__(self):
    CipherSolver.__init__(self)
    self._inherit_docs(CipherSolver)

We need to add a few bits of documentation to our existing methods inside the AristocratSolver class. Add the following doc strings:

  def frequency_list(self, length = 1, text = ""):
    """Displays counts for frequencies of characters"""

  def keys(self):
    """Displays the plaintext and ciphertext keys."""

  def set(self, ct, pt = "-"):
    """Sets the plaintext equivalent for each ciphertext character.
    You can enter multiple letters at a time.
    Enter a single dash '-' to set the plaintext characters to blank."""

PatristocratSolver (patristocrat.py)

PatristocratSolver class inherits from AristocratSolver so we need to call the _inherit_docs method in the PatristocratSolver.init:

class PatristocratSolver(AristocratSolver):
  def __init__(self):
    AristocratSolver.__init__(self)
    self._inherit_docs(AristocratSolver)

Conclusion

That was a lot! Hopefully, you survived up to this point. This concludes the documentation system for the interactive solver. You can add your own methods to any of the classes and add doc strings to document things. In Part V, the next and last article in this series, we will include loading the cons from the Cryptogram Digital Con files (available at cryptogram.org for members).

Complete Source Code

You can download the complete source code to Part IV at: Creating an interactive cryptogram solver (Part IV – Source).

Creating an interactive cryptogram solver (Introduction)
Creating an interactive cryptogram solver (Part I)
Creating an interactive cryptogram solver (Part II)
Creating an interactive cryptogram solver (Part III)

Creating an interactive cryptogram solver (Part III)

2009-09-23T00:38:28+00:00

In Part I and Part II of this series we created the framework for our interactive solver and finally had a working AristocratSolver class. In this part, we will enhance our existing framework by adding some commonly used functions, add frequency counting of characters and character sequences to the AristocratSolver class, add the ability to display the current plaintext and ciphertext keys to the AristocratSolver class and then finally create a PatristocratSolver class that reuses all our work in the AristocratSolver class.

Extending the Cipher Class

In the previous parts of this series, the Cipher class had two methods, encrypt and decrypt, that basically just returned the Cipher.text property. A problem that we will run into is that the Cipher.text property could contain characters that are not compatible with the cipher class that we are using. We’ll want to strip out any invalid characters before we had the text off to the encryption and decryption routines of the actual cipher class. First, lets import the shared class at the top of cipher.py so we can use it:

Contents of cipher.py (Place at the top of the file on line 3)

import shared

Now lets modify the _Cipher.__init__ method as follows:

Contents of cipher.py (Replace the existing Cipher.__init__ method)

  def __init__(self):
    self.text = ""
    self.decrypt_filter = lambda char: True
    self.encrypt_filter = lambda char: True

Here we’ve added self.decrypt_filter and self.encrypt_filter variables. These two variables will actually hold the functions that will get called for each character in Cipher.text and will return True or False depending on if the character is valid or not. We are using a really cool feature of python here called “lambda”. Lambda creates a single expression function. They are in the format:

lambda arguments: expression

We’ll see a better use of lambdas in the Aristocrat class in a moment. For now, the Cipher.decrypt_filter and Cipher.encrypt_filter just pass in each character and always return True. So the text will go unchanged because all characters are considered valid. Now lets add a method that will handle all the filtering of the text.

Contents of cipher.py (Insert after the Cipher.__init__ method)

  def get_text(self, text="", filter_func = None):
    if text == "":
      text = self.text
    if filter_func == None:
      return text
    else:
      filtered_text = ""
      for char in text:
        if filter_func(char):
          filtered_text += char
      return filtered_text

The Cipher.get_text method will use the Cipher.text property unless the text argument is passed. It then walks through the text and calls the filter_func for each character in the text. If filter_func returns True, the character is added to the return value. This is the perfect place to strip off line feeds or any other undesired character. The filter_func can be any function that has a character argument and returns True or False. Lets modify our Cipher.encrypt and Cipher.decrypt methods to take advantage of this new Cipher.get_text method.

Contents of cipher.py (Replace the existing Cipher methods)

  def decrypt(self, text=""):
    return self.get_text(text, self.decrypt_filter)

  def encrypt(self, text=""):
    return self.get_text(text, self.encrypt_filter)

The Cipher.encrypt and Cipher.decrypt methods are much simpler now because they just have to call the Cipher.get_text method with the appropriate filter. Alright, the Cipher class is now ready for other classes to utilize its new functionality! Now lets modify the CipherSolver to contain a few commonly used functions.

The CipherSolver Class

Contents of cipher.py (Replace the existing CipherSolver.__init__ method)

  def __init__(self):
    self.cipher = Cipher()
    self.prompt = ">"
    self.maxlinelen = 70
    self.shortcuts = {"d":"display"}

Here we’ve changed the CipherSolver.__init__ method by adding “self.maxlinelen = 70”. We want to add this here because we will be using the shared.breaklines function and will need to know what the maximum line length will be. This means that we can remove this line from the Aristocrat class if we want to. Now lets add a new method below the CipherSolver.display method.

Contents of cipher.py (Replace the existing CipherSolver.__init__ method)

  def print_counts(self,counts):
    retvalue = ""
    if len(counts) == 0:
      retvalue = "None"
    else:
      for item,count in counts:
        retvalue += "%s:%s, " % (item,count)
    print shared.breaklines(retvalue.strip(", "), self.maxlinelen)

A lot of our cipher classes are going to be calculating and counting things like frequency counts, digraph and trigraph counts, etc. So the CipherSolver.print_counts method neatly display those counts on the screen. All our cipher classes will be able to use this method and uniformly display their counts as needed

Shared functions

Before we jump into the Aristocrat class, lets add a few shared functions to our “shared.py” file. Add the imports to the top of the file and then add the two functions. I like to keep things in alphabetical order but that is totally up to you.

Contents of shared.py (Place at the top of the file)

import string
from operator import itemgetter

Contents of shared.py (ADD at the top of the file)

#Breaks the text into blocks of length blocklen
def breakblocks(text,blocklen):
  output = ""
  currlen = 0
  for index in range(len(text)):
    output += text[index]
    currlen += 1
    if currlen == blocklen:
      output += " "
      currlen = 0
  return output

The breakblocks will break a long line into chunks based on the blocklen parameter. This will be used in the PatristocratSolver class when we want to display things in 5 character blocks. Lets use the python interpreter and see how to use it:

python
>>> import shared
>>> line = 'GSRHRHZGVHGLUGSVVNVITVMXBYILZWXZHGHBHGVN'
>>> shared.breakblocks(line,5)
'GSRHR HZGVH GLUGS VVNVI TVMXB YILZW XZHGH BHGVN '
>>>

#Calculates how many times sequences of characters appears in the text
def calc_graphs(items, length, sorted = True, keepsingles = False, valid_chars = string.ascii_letters):
  if type(items) != list:
    items = [items]
  counts = {}
  for text in items:
    temp = ""
    for index in range(len(text)):
      if text[index] in valid_chars:
        temp = temp + text[index]
    text = temp
    for index in range(len(text)-(length-1)):
      graph = text[index:index+length]
      if not graph in counts:
        counts[graph] = 0
      counts[graph] += 1
  counts = counts.items()
  if not keepsingles:
    counts = [(k,v) for k,v in counts if v > 1]
  if sorted:
    counts.sort(key=itemgetter(1), reverse=True)
  return counts

The calc_graphs function takes a string or list of strings and calculates all the repeated sequences of characters that are of the length specified. If we want to throw away any that only appear once, we can pass in False for the keepsingles parameter. The valid_chars parameter is a string containing all the valid characters that will be counted. Lets look at an example of its usage:

python
>>> import shared
>>> line = 'GSRH RH Z GVHG LU GSV VNVITVMXB YILZWXZHG HBHGVN.'
>>> shared.calc_graphs(line,1)
\[('G', 6), ('H', 6), ('V', 6), ('Z', 3), ('B', 2), ('I', 2), ('L', 2), ('N', 2), ('S', 2), ('R', 2), ('X', 2)\]
>>>

If we want to see the digraphs we can pass in 2 as the length or 3 for trigraphs. By default the result is sorted descending from largest to smallest count.

The Aristocrat Class

Alright, now lets put all this new functionality into the Aristocrat class. Lets override the Cipher.encrypt_filter and Cipher.decrypt_filter properties in the Aristocrat.__init__ method.

Contents of aristocrat.py (Replace the existing Aristocrat.__init__ method)

  def __init__(self):
    Cipher.__init__(self)
    self.ctkey = string.ascii_uppercase
    self.ptkey = "-" * 26
    self.decrypt_filter = lambda char: char in (string.ascii_letters + string.punctuation + " ")
    self.encrypt_filter = self.decrypt_filter

Here we are setting the decrypt_filter to a new lambda function. This function checks to see if the character passed is a letter, punctuation or a space. We want the encrypt_filter to be the same so we are just assigning it to the value of the decrypt_filter. Now, whenever we need to get the text in the Aristocrat class we will strip out any line feeds, digits or other invalid characters automatically.

The AristocratSolver Class

Now lets add our new frequency counting functionality to the AristocratSolver class.

Contents of aristocrat.py (Replace the existing AristocratSolver.__init__ method)

  def __init__(self):
    CipherSolver.__init__(self)
    self.cipher = Aristocrat()
    self.shortcuts['f'] = 'frequency_list'
    self.shortcuts['k'] = 'keys'
    self.shortcuts['s'] = "set"

Here we’ll add two new shortcuts to the Cipher.shortcuts list in the AristocratSolver.__init__ method for the frequency_list method and the keys method. So whenever we want to get frequency counts, we just have to type “f”. If you want digraphs just type “f 2” into the solver. If we want to display the keys we can just type “k”. Notice we’ve removed the “self.maxlinelen = 70” line from the AristocratSolver.__init__ method. Since the CipherSolver.__init__ method already declares this, we don’t need to do this. If we want to change the maxlinelen globally, we can just change it in CipherSolver.

The last thing we need to do is add the actually frequency_list method to the AristocratSolver class below the display method.

Contents of aristocrat.py (Insert after the AristocratSolver.display method)

  def frequency_list(self, length = 1, text = ""):
    text = Cipher.encrypt(self.cipher, text)
    self.print_counts(shared.calc_graphs(text.split(" "), int(length)))

This method just gets the Cipher.text property and uses the shared.calc_graphs and print_counts functions to calculate and display the frequency counts. Notice that the text that we are passing into the shared.calc_graphs function is actually being split into chunks at the spaces. This makes sure that we only see repeated patterns in the actual words themselves and not just mashed together by ignoring the spaces

Lets see this new method in action! Lets fire up our “sandbox.py” in python:

python sandbox.py
> d

GSRH RH Z GVHG LU GSV VNVITVMXB YILZWXZHG HBHGVN.
---- -- - ---- -- --- --------- --------- ------.

> f 1

G:6, H:6, V:6, Z:3, B:2, I:2, L:2, N:2, S:2, R:2, X:2

> f 2

HG:3, GV:2, GS:2, VN:2, RH:2

>

Now lets add the displaying of plaintext and ciphertext keys to the AristocratSolver class.

Contents of aristocrat.py (Insert after the AristocratSolver.frequency_list method)

  def keys(self):
    print "ct:", self.cipher.ctkey, "npt:", self.cipher.ptkey, "n"
    values = zip(self.cipher.ptkey, self.cipher.ctkey)
    values.sort()
    ptval=""
    ctval=""
    for pt,ct in values:
      ptval += pt
      ctval += ct
    print "pt:",ptval, "nct:", ctval

The key function displays the ciphertext and plaintext keys, first in alphabetical ciphertext order and then alphabetical plaintext order. Lets see what it looks like when executed:

python sandbox.py
> s abcdefghijklmnopqrstuvwxyz zyxwvutsrqponmlkjihgfedcba

GSRH RH Z GVHG LU GSV VNVITVMXB YILZWXZHG HBHGVN.
this is a test of the emergency broadcast system.

> k

ct: ABCDEFGHIJKLMNOPQRSTUVWXYZ
pt: zyxwvutsrqponmlkjihgfedcba

pt: abcdefghijklmnopqrstuvwxyz
ct: ZYXWVUTSRQPONMLKJIHGFEDCBA

>

There we go! We’ve extended our AristocratSolver class so that it can display frequency counts and keys. We’ve modified our Cipher and CipherSolver classes to make them more extendable for other cipher classes. The last thing we need to do is whip up the Patristocrat and PatristocratSolver classes.

The Patristocrat Class

Since Patristocrats are basically just Aristocrats without the word spacings, we can re-use a ton of the Aristocrat class functionality and only override the different parts. Lets create a new file called “patristocrat.py” and put in the following code:

Contents of patristocrat.py

import string
import shared
from cipher import Cipher
from aristocrat import Aristocrat, AristocratSolver

class Patristocrat(Aristocrat):
  def __init__(self):
    Aristocrat.__init__(self)
    self.decrypt_filter = lambda char: char in string.ascii_letters
    self.encrypt_filter = self.decrypt_filter

The Patristocrat class is quite small because the Aristocrat class already does everything. Notice we are inheriting from the Aristocrat class and calling the Aristocrat.__init__ method. The Aristocrat class allowed letters, punctuation, and spaces in both its encrypted and decrypted text. For Patristocrats, we really only want letters. So we change the decrypt_filter and encrypt_filter to only return True if the character is a letter (A-Z, a-z).

The PatristocratSolver class

The PatristocratSolver class has a little more code than its cipher.

Contents of patristocrat.py (Place after the Patristocrat class)

class PatristocratSolver(AristocratSolver):
  def __init__(self):
    AristocratSolver.__init__(self)
    self.cipher = Patristocrat()

  def display(self):
    data = ""
    ct = shared.breakblocks(Cipher.encrypt(self.cipher), 5)
    ct = shared.breaklines(ct, self.maxlinelen).split("n")
    pt = shared.breakblocks(self.cipher.decrypt(), 5)
    pt = shared.breaklines(pt, self.maxlinelen).split("n")

    for index in range(len(ct)):
      data += ct[index] + "n"
      data += pt[index] + "nn"
    print data.strip("n")

  def frequency_list(self, length = 1, text = ""):
    text = Cipher.encrypt(self.cipher, text)
    self.print_counts(shared.calc_graphs(text, int(length)))

Lets go through each of the three methods in the class

  def __init__(self):
    AristocratSolver.__init__(self)
    self.cipher = Patristocrat()

First we have the PatristocratSolver.__init__ method. This calls the AristocratSolver.__init__ method so it can inherit all its functionality. We then change the self.cipher to be a new instance of the Patristocrat class.

  def display(self):
    data = ""
    ct = shared.breakblocks(Cipher.encrypt(self.cipher), 5)
    ct = shared.breaklines(ct, self.maxlinelen).split("n")
    pt = shared.breakblocks(self.cipher.decrypt(), 5)
    pt = shared.breaklines(pt, self.maxlinelen).split("n")

    for index in range(len(ct)):
      data += ct[index] + "n"
      data += pt[index] + "nn"
    print data.strip("n")

The display method gets the text from the Cipher.encrypt method and then breaks it into five character blocks. We use the shared.breaklines method again to break the lines if they are too long. We also break the decrypted text into blocks and then display the ciphertext and plaintext together.

  def frequency_list(self, length = 1, text = ""):
    text = Cipher.encrypt(self.cipher, text)
    self.print_counts(shared.calc_graphs(text, int(length), False))

Finally, we have the frequency_list method. The only thing that is different from the AristocratSolver.frequency_list is that we are not splitting the text at the spaces or line breaks.

That is it for the Patristocrat and PatristocratSolver classes. We saved a lot of work by inheriting from the Aristocrat classes. Now lets modify “sandbox.py” and take her for a spin.

Contents of sandbox.py (Replace existing contents)

from patristocrat import PatristocratSolver
solver = PatristocratSolver()
solver.cipher.text = 'GSRHRHZGVH GLUGSVVNVITVMXBYIL ZWXZHGHBHGVN'
solver.solve()

So we just set everything to use the Patristocrat classes instead. I’ve thrown a little ciphertext in there with spaces so you can see that they will be stripped out by the filters. Lets see how this looks in python:

\> d

GSRHR HZGVH GLUGS VVNVI TVMXB YILZW XZHGH BHGVN
----- ----- ----- ----- ----- ----- ----- -----

> f

G:6, H:6, V:6, Z:3, B:2, I:2, L:2, N:2, S:2, R:2, X:2

> s VNVITVMXB emergency

GSRHR HZGVH GLUGS VVNVI TVMXB YILZW XZHGH BHGVN
----- ---e- ----- eemer gency -r--- c---- y--em

> k

ct: ABCDEFGHIJKLMNOPQRSTUVWXYZ
pt: -y------r---nm-----g-e-c--

pt: -------------------cegmnry
ct: ACDEFGHJKLOPQRSUWYZXVTNMIB

>

It works! You can still use all the AristocratSolver methods like keys. Our framework has been completed and we’ve seen how to create two different cipher classes and interactive solvers. They can be extended to fit any solving style or cipher type. In Part IV of this series we will bring all the code together into a complete interactive solver that allows you to select which solver class you would like to use. We will also add documentation functionality to the classes so that the solver can tell you all about the functions and parameters without us having to constantly look at the source code or memorize everything.

Complete Source Code

You can download the complete source code to Part III at: Creating an interactive cryptogram solver (Part III – Source).

Code Penguin

Custom Polyglot Notebook Kernel

Calculator Kernel and Infrastructure

Calculator Class

Handling the SubmitCode command

Testing the Calculator Kernel

Variables

Accessing External Variables

Exposing Internal Variables

Auto Completion

Hover Text

Nuget Packages

Conclusion

OpenID Connect - Dotnet Desktop Application

Configuring the Authentication Server

Auth0 User Roles

Create the Dotnet Console Application

AuthenticationSettings Class

AuthorizationCallbackResponse Class

Worker.cs Class

ExecuteAsync Method

AuthenticateExternalUserAsync Function

RetrieveOpenIdConnectConfigurationAsync Function

RequestAuthorizationCodeAsync Function

BuildUrl Function

OpenSystemBrowser Method

RequestTokenAsync Function

ValidateToken Function

Retrieving the Authorization Code

ICallbackListener Interface

CallbackListenerFactory Class

HttpCallbackListener Class

Listen Method

GetRedirectUriPort Function

WaitForResponseAsync

Testing out the console application

Microsoft Identity Platform

Google’s OAuth 2.0 authentication system

Conclusion

ShortDash: Cross-platform Shortcut Dashboard

Outwitters Sports Network

Outwitters Replay Viewer

Outwitters Sports Network is born!

Technical Details

Summary

Slipix Version 1.0 for iOS

Introduction

Video

Screen Shots

WebSolver: An Interactive Cryptogram Solver for the Web!

Quick Overview

Useful commands

Save and Continue

Command History

Conclusion

Technical Details and Links

Pattern Words: A Solver’s Staple

Pattern Types

Fixed Length Patterns

Floating Length Patterns

High/Low Frequency Patterns

Searching for Pattern Words

Conclusion

Beginner’s Guide to the ACA

Creating an interactive cryptogram solver (Part IV)

Clearing the screen

The Solving Gateway: solver.py

The Documentation System

Solver Classes

Methods

Properties

Document Everything!

CipherSolver (cipher.py)

AristocratSolver (aristocrat.py)

PatristocratSolver (patristocrat.py)

Conclusion

Complete Source Code

Related Posts:

Creating an interactive cryptogram solver (Part III)

Extending the Cipher Class