• Adding our first feature to Ionide

    This post is part of the F# Advent Calendar in English 2017 organized by Sergey Tihon

    Ionide is now one of the most used F# editing experience but it's pretty unique as it's based on an easily extensible editors: Visual Studio Code.

    In the world of VSCode extensibility no arcane knowledge is required, the editor is open source with a well written code base (in TypeScript), the plugin API is easy to get into and most plugins are open source too. While the Javascript (Or TypeScript) language might not be to the taste of everyone, we have Fable allowing us to continue to write F# and run in in the Javascript VM.

    And Ionide is written in Fable, so we'll use Ionide and VSCode to edit Ionide itself.

    Let's implement a very simple feature and then move to a bigger one! The simple feature will be to add a button in the editor when .fsx files are edited to call the FSI: Send File command, sending the full file to F# interactive. The more complex one will be to run the file in the terminal.

    Getting started

    First you'll need the prerequisites (VScode, F#, dotnet, yarn, node, ...) and then let's checkout a commit without the feature, run a first full build and start code :

    git clone git@github.com:ionide/ionide-vscode-fsharp.git
    cd ionide-vscode-fsharp
    git checkout -b having_fun c0922fc
    build build
    code .
    

    Once this first full build is setup well start our developer loop in VS Code:

    • In the Integrated terminal (View > Integrated Terminal menu) we run build watch :
      Build watch running
    • In the debug view we select the Launch Only configuration:
      Launch only being selected
    • Press F5 to start a debug instance of VS Code.

    Once started our loop is simple :

    • Type new buggy code.
    • Wait for the terminal to show that it rebuilt it successfully
    • Move to the debug instance and press F1 to run Reload Window
    • Test
    • Repeat

    The loop is fast and while not as good as a browser with auto-reloading, pretty nice to use.

    A JSON-only feature

    Our first target is to get something like that:

    Final result

    And we actually don't need any code to do it, as the command already exists the only thing we need to do is to to change the release/package.json file. And as added bonus it's not something that is build but instead used as-is by Code so we don't even need to wait for the watch build to test it, simply reloading the test instance window will allow us to see our changes.

    While the file contains the same field as any package.json file, it's a section specific to Code that interest us : contributes. It contains all of the different things that the Ionide extensions declare that it's able to provide and it's well documented on VS Code website.

    The command we want to run is shown as FSI: Send File in the command palette but to run it we'll need it's ID and that can be found in the commands section.

    {
      "contributes": {
        "commands": [
          {
            "command": "fsi.SendFile",
            "title": "FSI: Send File"
          }
        ]
      }
    }
    

    We currently want to show it in the editor title menu bar that is documented as being the editor/title section inside menu, so let's add it:

    {
      "contributes": {
        "menu": {
            "editor/title": [
              {
                "command": "fsi.SendFile",
                "when": "editorLangId == 'fsharp' && resourceExtname == '.fsx'",
                "group": "navigation"
              }
            ],
        }
      }
    }
    

    A few notes here:

    • command is the command we wants to run, that we found just before
    • when is the condition for it to appear, we want to be sure that the file is marked as F# (The language ID is auto detected but the user can also change it manually via the selector on the bottom right) and that it's a script (Sending .fs files is possible but we don't want to suggest it so prominently in the UI)
    • group is used to group related commands but navigation is special as it tell code to make the item visible directly as an icon instead of being a line in the ... menu (the default)

    Note: resourceExtname is a feature that was added to VS Code by Krzysztof Cieślak, the author of Ionide specifically for this change!

    We reload the window and...

    No icon

    We forgot the icon 😢

    Turns out that the icon is declared along with the command (And show everywhere the command is shown) instead of being part of the menu, so we'll get back to the command definition and add the icon

    {
      "contributes": {
        "commands": [
          {
            "command": "fsi.SendFile",
            "title": "FSI: Send File",
            "icon":
            {
              "light": "./images/send-light.svg",
              "dark": "./images/send-dark.svg"
            }
          }
        ]
      }
    }
    

    The images need to be placed in the release/image folder and to start copying existing ones is good enough. There are 2 images to allow for small variations between light and dark themes, but icon can also be the path to a single image if the distinction isn't needed.

    And finally our feature is here, we can test it and verify that it works as expected, job done 🙌.

    Final result

    Adding our own command

    Now let's add another small feature: Running FSX scripts in the terminal

    Menu with run and send

    To do that well go back to package.json and add our new command:

    {
      "contributes": {
        "commands": [
          {
            "command": "fsharp.scriptrunner.run",
            "title": "F#: Run script",
            "icon":
            {
              "light": "./images/run-light.svg",
              "dark": "./images/run-dark.svg"
            }
          }
        ],
        "menus": {
          "commandPalette": [
            {
              "command": "fsharp.scriptrunner.run",
              "when": "editorLangId == 'fsharp' && resourceExtname == '.fsx'"
            }
          ],
           "editor/title": [
            {
              "command": "fsharp.scriptrunner.run",
              "when": "editorLangId == 'fsharp' && resourceExtname == '.fsx'",
              "group": "navigation"
              }
            ]
          ]
        }
      }
    }
    

    We'll also need to add a new file to host our code in src/Components/ScriptRunner.fs and add it to the .fsproj file. (After this step the watch script might need to be restarted manually to pickup the new file)

    namespace Ionide.VSCode.FSharp
    
    open System
    open Fable.Core
    open Fable.Import.vscode
    open Fable.Import.Node
    
    module ScriptRunner =
        let private runFile () =
            printfn "Hello world"
    
        let activate (context: ExtensionContext) =
            commands.registerCommand(
                "fsharp.scriptrunner.run",
                runFile |> unbox<Func<obj,obj>>) |> context.subscriptions.Add
    

    Adding the call to activate in fsharp.fsx is the final touch and our new button will finally do something (Even if it's only writing "Hello world" to the Debug Console)

    // ...
    Forge.activate context
    Fsi.activate context // <--- Line added here
    ScriptRunner.activate context
    
    { ProjectLoadedEvent = Project.projectLoaded.event
      BuildProject = MSBuild.buildProjectPath "Build"
      GetProjectLauncher = Project.getLauncher
      DebugProject = debugProject }
    

    Finally, we write some code

    Let's start with the most simple addition, creating a terminal with a fixed name that run fsi.exe with the current file as single parameter

    let private runFile () =
        let scriptFile = window.activeTextEditor.document.fileName
        let terminal = window.createTerminal("script", Environment.fsi, [| scriptFile|])
        terminal.show ()
    

    It works but show a big problem with our approach as it closes immediately the terminal, before we can even read the results of executing our script.

    A solution to this problem is to run a shell (cmd.exe for windows) and use it's capabilities instead of directly starting F# Interactive.

    Ideally we would use it's arguments that allow to run commands but windows program arguments function differently than unix platforms: They are a single string and programs are free to parse them as they want. As NodeJS started on unix it's API wants an array of parameters that are then encoded using the MSVCRT rules but cmd.exe parameter /C has a very special handling that doesn't follow theses conventions and we can't use it.

    But VSCode allow us to send simulated keystrokes to the terminal so we'll use this

    let private runFile () =
        let scriptFile = window.activeTextEditor.document.fileName
        let terminal = window.createTerminal("script", "cmd.exe", [||])
        terminal.sendText(sprintf "\"%s\" \"%s\" && pause && exit" Environment.fsi scriptFile)
        terminal.show ()
    

    First try, working

    We now have a working minimal sample and can start to build from there:

    • We need to change our directory to the one of the script (the default is the root of the workspace) before running it
    • The fixed title become confusing pretty fast if we run multiple different scripts

    For both we might be tempted to use System.IO, but that's not something that's currently translated:

    ERROR in ./src/Components/ScriptRunner.fs
    G:/Code/_ext/ionide-fs/src/Components/ScriptRunner.fs(24,24): (24,67) error FABLE: Cannot find replacement for System.IO.Path::GetDirectoryName
     @ ./src/fsharp.fs 45:0-71
     @ ./src/Ionide.FSharp.fsproj
    

    The solution is to use the NodeJS Path API directly, the whole NodeJS API is provided by Fable in the Fable.Import.Node Nuget package that Ionide already import

    open Fable.Import.Node
    
    let private runFile () =
        let scriptFile = window.activeTextEditor.document.fileName
        let scriptDir = Path.dirname(scriptFile)
    
        let title = Path.basename scriptFile
        let terminal = window.createTerminal(title, "cmd.exe", [| "/K" |])
        terminal.sendText(sprintf "cd \"%s\" && \"%s\" \"%s\" && pause && exit" scriptDir Environment.fsi scriptFile)
        terminal.show ()
    

    Now let's cleanup, add unix support and we're ready to send a PR:

    namespace Ionide.VSCode.FSharp
    
    open System
    open Fable.Import.vscode
    open Fable.Import.Node
    
    module ScriptRunner =
        let private runFile (context: ExtensionContext) () =
            let scriptFile = window.activeTextEditor.document.fileName
            let scriptDir = Path.dirname(scriptFile)
    
            let (shellCmd, shellArgs, textToSend) =
                match Os.``type``() with
                | "Windows_NT" ->
                    ("cmd.exe",
                     [| "/Q"; "/K" |],
                     sprintf "cd \"%s\" && cls && \"%s\" \"%s\" && pause && exit" scriptDir Environment.fsi scriptFile)
                | _ ->
                    ("sh",
                     [||],
                     sprintf "cd \"%s\" && clear && \"%s\" \"%s\" && echo \"Press enter to close script...\" && read && exit" scriptDir Environment.fsi scriptFile)
    
            let title = Path.basename scriptFile
            let terminal = window.createTerminal(title, shellCmd, shellArgs)
            terminal.sendText(textToSend)
            terminal.show ()
    
        let activate (context: ExtensionContext) =
            commands.registerCommand(
                "fsharp.scriptrunner.run",
                runFile |> unbox<Func<obj,obj>>) |> context.subscriptions.Add
    

    Conclusion

    VS Code an Ionide are the perfect introduction for any programmer that wants to start customizing his tools so don't hesitate.

    You're missing an icon ? Some command would make your life easier ? Fork, Build, Contribute !

  • More F# colors in the terminal

    After releasing ColoredPrintf I started investigating color terminals, especially how some terminals can display more than the traditional 16 colors.

    While on windows console colors are changed using a call to the SetConsoleTextAttribute API, on unix XTerm and other terminals follow the way the VT100 physical terminal did it with special escape sequences that are sent to the terminal but change it's behavior instead of being displayed.

    It is especially interesting as such terminals are becomming more present, even in the windows world. ConEmu support a lot even if it's buggy, the venerable windows terminal can be told to interpret them and it'll even soon gain True Color support too.

    Compatibility

    If you're on windows, expect bugs everywhere but if you have a mac or a linux OS you're good. In more details :

    • On windows ConEmu is still the best you can do, it's buggy but usable. There is a little trick to enable more than 16 colors but it's easy and need to be done only once per session, I'll explain how to do it from code in Appendix A but you can simply display one of the ConEmu sample file with cmd /c type "%ConEmuBaseDir%\Addons\AnsiColors256.ans" and it'll activate it.
    • On macOS (Tested on sierra) the integrated terminal can display 256 colors without problems but will translate any True Color command to 16 colors, giving a very ugly result. But iTerm2 is free works !
    • On linux the Gnome default terminal, XTerm and Konsole works flawlessly (XTerm adapt true color codes to 256 colors but it's algorithm is pretty good).
    • Visual Studio Code integrated terminal can do most of them and as the REPL provided by the Ionide plugin uses it we can use the F# REPL directly there. But beware the windows version of the VSCode terminal is pretty buggy at the moment and often completely break when control codes are sent (Well, resizing it is enough to break it, you don't need much 😁)

    Using F# to play with escape sequences

    Escape sequences are generated by printing the ESC (0x1B) character followed by at least another character.

    The most common ones start with the "CSI" sequence ESC[ followed by parameters and ending with a character designating the command. The full list is on Wikipedia or MSDN.

    Let's try that in F# :

    let esc = string (char 0x1B)
    System.Console.WriteLine(esc + "[31;1m" + "Hello world" + esc + "[0m")
    

    First sample

    It displayed Hello World in bright red as m is the graphic command taking sub-commands separated by ;, 31 is the red color and 1 signal that we want bold/bright text. And at the end 0 is the reset sub-command, without it the terminal would stay red on all following lines.

    Using kprintf we can create a printf-like function and start implementing a few of the escape sequences :

    let csi = esc + "["
    let printsequencef f = Printf.kprintf (fun s -> System.Console.Write(esc + s)) f
    let printcsif f = Printf.kprintf (fun s -> System.Console.Write(csi + s)) f
    let selectGraphicRendition (gr: int list) =
        printcsif "%sm" (System.String.Join(";", gr |> Seq.map string))
    let resetColor() = selectGraphicRendition [0]
    let setForeground i = selectGraphicRendition([30 + i])
    let setBackground i = selectGraphicRendition([40 + i])
    

    256 colors is enough for everyone

    Until then we've done nothing very interesting, after all the .Net console already support changing the console color to bright red, nothing new under the sun. But what the .Net console or Windows native API doesn't support is the 38 and 48 graphics sub-commands, extended foreground an background colors.

    let setExtendedForeground i = selectGraphicRendition [38; 5; i]
    let setExtendedBackground i = selectGraphicRendition [48; 5; i]
    let setForegroundRgb r g b = selectGraphicRendition [38; 2; r; g; b]
    let setBackgroundRgb r g b = selectGraphicRendition [48; 2; r; g; b]
    

    The first variant accept an index in a 256 colors table and the second specify the R, G and B components.

    While all the terminal emulators I tested support both of them the colors the display will vary, especially for the R;G;B True Color mode. Some like macOS default terminal are pretty weird in the sense that it support 256 color mode but interpolate any R;G;B code to 16 colors.

    Let's see what the palete looks like :

    let extendedBlock i =
        for j in i..i+5 do
            setExtendedBackground j
            printf "  "
    
    for row in 0..5 do
        for b in 0..5 do
            extendedBlock (16 + 36*b + row*6)
            resetColor()
            printf " "
        printfn ""
    

    The result in VS Code (On macOS) :

    Palete

    We need to be true to our colors

    Now that we know how to draw with true colors we can draw pictures on the terminal :

    open System.Drawing
    
    let showBitmap path =
        let bitmap = Image.FromFile(path) :?> Bitmap
        for y in 0..bitmap.Size.Height-1 do
            for x in 0..bitmap.Size.Width-1 do
                let px = bitmap.GetPixel(x, y)
                setBackgroundRgb (int px.R) (int px.G) (int px.B)
                printf "  "
            resetColor()
            printfn ""
    

    And we can apply it to a 40x40px version of the FSharp logo :

    Logo

    The code for all of that can be found in a Gist.

    I didn't test the windows insider release so can't tell how well microsoft True Color support in the native console works but it's the next thing I want to try.

    Appendix A: ConEmu color support

    ConEmu support more than the standard 16 colors but if you try it, it won't work directly, it seem that maybe for compatibility reasons the support isn't always on.

    Before

    To trigger it an application must first scroll the terminal far away at least once with the T command :

    let scrollUp (lines: int) = printcsif "%iS" lines
    let scrollDown (lines: int) = printcsif "%iT" lines
    scrollDown 9999
    setCursorPos 9999 1
    

    After that ConEnu will display what we expect :

    After

    Appendix B: Enabling escape codes in Windows default terminal

    While the windows terminal support escape codes, it doesn't do it directly. They need to be enabled using SetConsoleMode API :

    #nowarn "9"
    module DllImports =
        open System.Runtime.InteropServices
        open Microsoft.FSharp.NativeInterop
    
        let INVALID_HANDLE_VALUE = nativeint -1
        let STD_OUTPUT_HANDLE = -11
        let ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x0004
    
        [<DllImport("Kernel32")>]
        extern void* GetStdHandle(int nStdHandle)
    
        [<DllImport("Kernel32")>]
        extern bool GetConsoleMode(void* hConsoleHandle, int* lpMode)
    
        [<DllImport("Kernel32")>]
        extern bool SetConsoleMode(void* hConsoleHandle, int lpMode)
    
        let enableVTMode() =
            let handle = GetStdHandle(STD_OUTPUT_HANDLE)
            if handle <> INVALID_HANDLE_VALUE then
                let mode = NativePtr.stackalloc<int> 1
                if GetConsoleMode(handle, mode) then
                    let value = NativePtr.read mode
                    let value = value ||| ENABLE_VIRTUAL_TERMINAL_PROCESSING
                    SetConsoleMode(handle, value)
                else
                    printfn "no get"
                    false
            else
                printfn "no handle"
                false
    
    DllImports.enableVTMode() |> ignore
    
  • ColoredPrintf released

    ColoredPrintf is a very simple library that provide a printf-like function with color support. A first version of the library is available on NuGet with it's source code on my Github.

    The syntax to use for color is $foreground;background[text] with the color names being the same as in the ConsoleColor enumeration.

    The main entry points of the library are the colorprintf and colorprintfn functions that have the same signature as printf and printfn (It uses my MasterOfFoo library for that).

    Example

    colorprintfn "Hello $red[world]."
    colorprintfn "Hello $green[%s]." "user"
    colorprintfn "$white[Progress]: $yellow[%.2f%%] (Eta $yellow[%i] minutes)" 42.33 5
    colorprintfn "$white;blue[%s ]$black;white[%s ]$white;red[%s]" "La vie" "est" "belle"
    

    result

  • First release for MasterOfFoo

    MasterOfFoo is an F# library that intend to facilitate writing functions that act like the native printf one does.

    I presented it as an experiment during september's Paris F# Meetup (See the summary in french) but I feel like it's now ready to be tested by more people.

    It is now available on nuget as a preview version of 0.1.2 and the source code is on GitHub.

    The API is described in more details in the readme but here is a small taste of the possibilities with a proof-of-concept sqlCommandf implementation :

    module SqlCommandBuilder =
        open System.Text
        open BlackFox.MasterOfFoo
        open System.Data.Common
        open System.Data.SqlClient
    
        type internal SqlEnv<'Result, 'cmd when 'cmd :> DbCommand>(command: 'cmd, k) =
            inherit PrintfEnv<unit, unit, 'Result>(())
            let queryString = StringBuilder()
            let mutable index = 0
    
            override __.Finalize(): 'Result =
                command.CommandText <- queryString.ToString()
                k(command)
    
            override __.Write(s : PrintableElement) =
                let asPrintf = s.FormatAsPrintF()
                match s.ElementType with
                | PrintableElementType.FromFormatSpecifier ->
                    let parameter =
                        if typeof<DbParameter>.IsAssignableFrom(s.ValueType) then
                            s.Value :?> DbParameter
                        else
                            let paramName = sprintf "@p%i" index
                            index <- index + 1
    
                            let parameter = command.CreateParameter()
                            parameter.ParameterName <- paramName
                            parameter.Value <- s.Value
                            parameter
    
                    ignore(queryString.Append parameter.ParameterName)
                    command.Parameters.Add parameter |> ignore
                | _ ->
                    ignore(queryString.Append asPrintf)
    
            override __.WriteT(()) = ()
    
        let sqlCommandf (format : Format<'T, unit, unit, SqlCommand>) =
            doPrintfFromEnv format (SqlEnv(new SqlCommand (), id))
    
        let sqlRunf (connection: SqlConnection) (format : Format<'T, unit, unit, unit>) =
            let execute (cmd: SqlCommand) = cmd.ExecuteNonQuery() |> ignore
            doPrintfFromEnv format (SqlEnv(connection.CreateCommand(), execute))
    
    // Create the parametrized query "SELECT * FROM users WHERE name=@p0"
    let cmd = sqlCommandf "SELECT * FROM users WHERE name=%s" "vbfox"
    
    sqlRunf connection "DELETE FROM users WHERE name=%s" "vbfox"
    
  • Suave in IIS - Hello world

    Suave is an open source F# library implementing a web server in a functional style. The source code is on GitHub and releases are available on NuGet like most .NET open source projects these days.

    You can find the full sample presented here as a Visual Studio 2015 project on my GitHub. I'm using Paket to get the Suave package but the standard NuGet UI would work as well (albeit slower).

    The minimal hello world is simply :

    open Suave
    
    [<EntryPoint>]
    let main argv =
        startWebServer defaultConfig (Successful.OK "Hello World!")
        0
    

    startWebServer is a bloking function that take two parameters:

    • A SuaveConfig object defining the configuration, the default one opens a single non encrypted server on port 8083.
    • A WebPart representing everything that would be served. The type is actually an alias for HttpContext -> Async<HttpContext option> and suave provide a lot of filters & combinators to do standard things, like for example returning something different depending on the URL (Kinda expected for a web server).

    Maybe i'll blog more about the different combinators and how to combine them to reproduce a few common web server cases (Serving files, protecting access to some pages, ...) but as this post is about IIS integration i'll keep the hello world code and build from that.

    IIS

    IIS is the Microsoft web server present by default on Windows Server & Desktops. If you don't already have it on your dev machine it's available from the Turn Windows features on or off in control panel. The entry is named Internet Information Services and you can either go with the defaults or get everything under there.

    Windows features

    Once installed, start IIS Manager and create a website listening on a random port on IP address 127.0.0.1 with a Physical path corresponding to the bin/Debug directory of the hello world project.

    HttpPlatformHandler

    HttpPlatformHandler is an IIS addon that allow to use any http serving application where the executable is able to receive the port to listen to via an environment variable.

    To start the web site IIS run the executable with the HTTP_PLATFORM_PORT environment variable containing the port that it need to listen to and the module will then proxy requests to this port and sent back the response to the client. As the module insert itself inside the standard IIS processing it mean that things like logging, https certificates, respawning the process when an error happens or doing windows domain authentication can be handled by IIS without any change to the application code.

    To install it start the Web Platform Installer (Either from your start menu or from the right side actions in IIS Manager) and search for it.

    Web Platform Installer

    To be able to enable it per-site like we'll do in the next step, open IIS Manager and in the Feature View for the whole server, open Feature Delegation and enable Handlers Mappings as Read/Write.

    Using web.config

    IIS expect it's configuration in an XML file named web.config so you'll need to add an xml file named like that to your project with Copy to Output Directory configured to Copy if newer with the content :

    <?xml version="1.0" encoding="UTF-8"?>
    <configuration>
        <system.webServer>
            <handlers>
                <remove name="httpplatformhandler" />
                <add name="httpplatformhandler"
                    path="*"
                    verb="*"
                    modules="httpPlatformHandler"
                    resourceType="Unspecified" />
            </handlers>
            <httpPlatform processPath="./SuaveIIS.exe"
                arguments=""
                stdoutLogEnabled="true"
                stdoutLogFile="./SuaveIIS.log"
                startupTimeLimit="20"
                requestTimeout="00:05:00">
            </httpPlatform>
        </system.webServer>
    </configuration>
    

    This assume that your exectutable is named SuaveIIS.exe so adapt accordingly.

    For this to work you'll also need to authorize IIS to read it, do to that open Explorer and change the Permissions on the bin folder to allow read & write access by the local IIS_IUSRS group.

    Getting the port to listen to from F# Code

    In the current state, HttpPlatformHandler will start your executable, kill it after 20s and restart it in a loop because it will never see it listening on it's specified port so we need to read the port from the environment variable :

    module IISHelpers =
        open System
    
        /// Port specified by IIS HttpPlatformHandler
        let httpPlatformPort =
            match Environment.GetEnvironmentVariable("HTTP_PLATFORM_PORT") with
            | null -> None
            | value ->
                match Int32.TryParse(value) with
                | true, value -> Some value
                | false, _ -> None
    

    And then when we detect that we're started from IIS (Because IISHelpers.httpPlatformPort is Some) we can configure Suave to listen locally on that port and nowhere else :

    open Suave
    
    [<EntryPoint>]
    let main argv =
        let config = defaultConfig
        let config =
            match IISHelpers.httpPlatformPort with
            | Some port ->
                { config with
                    bindings = [ HttpBinding.mkSimple HTTP "127.0.0.1" port ] }
            | None -> config
    
        startWebServer config (Successful.OK "Hello World!")
        0
    

    If you compile it and visit your IIS site in a browser you should see your Hello World being served by IIS.

    Remarks

    • If you can't compile because the executable file is already it mean that your executable is running in IIS. What you need to do is to remove the web.config file from the bin/Debug directory, IIS will take this as a hint that your configuration changed and that it need to stop using your binary.
  • Ode to LINQPad

    LINQPad is one of my favorite tools for C# development it allow for fast development of small bits of C# to prototype or answer StackOverflow questions.

    LINQPad UI

    The original feature was to allow easy scripting of databases by providing a LINQ to SQL view of all objects of a connection directly as members of the generated script class. But LINQPad is a lot more generic and can serve as a C# swiss army knife for a lot of utility scripts.

    Searching for NuGet packages

    The Free and Pro editions of LINQPad don't give you access to the ability to reference NuGet packages directly and the NuGet website doesn't provide any download link. We'll use LINQPad itself to be able to search and download any package by Id.

    The first thing we need to do is to get access to the list of packages. If possible not in the HTML tag-soup form...

    Turns out that NuGet API is partially OData-XML and partially some REST endpoint. As LINQPad natively support OData I'll use only that. To use it, add a new connection of type WCF Data Services pointing to https://www.nuget.org/api/v2/ and then create a C# Program query using this connection.

    To start exploring we'll select some data to look at what fields are available and how they look:

    void Main()
    {
        Packages.Take(10).Dump();
    }
    

    Dump is a method that display an object in the output with a graphical representation. For tabular data (IQueryable, IEnumerable, ...) passing true as first argument will show it in a DataGrid.

    To search for a package we will want to filter by Id and get only the latest non-prerelease version so the fields we are interested in are IsAbsoluteLatestVersion and IsPrerelease in addition to Id :

    void Main()
    {
        var search = Util.ReadLine("Id");
        var packages =
            from package in Packages
            where package.IsAbsoluteLatestVersion
                && !package.IsPrerelease
                && package.Id.StartsWith(search)
            select package;
        packages.Dump(true);
    }
    

    This version of the code also introduce Util.ReadLine a method that display an input line at the bottom of the LINQPad output.

    Adding download links

    Now that we have all info from our package we need the download URL, turns out that it isn't directly included in the data but we should be able to build it from the information we already have.

    As NuGet is open-source we can read the source code and find how package download urls are constructed :

    // Route to get packages
    routes.MapDelegate("DownloadPackage",
                       "api/v2/package/{packageId}/{version}",
                       new { httpMethod = new HttpMethodConstraint("GET") },
                       context => CreatePackageService().DownloadPackage(context.HttpContext));
    

    We know version and packageId so we can now display the download URL.

    But LINQPad allow us to do a lot more: the Hyperlinq type is a magic one that when passed to Dump show a clickable link in the output window.

    The link can be constructed with an Action delegate that will be executed when clicked but for such a simple case we'll use the version taking directly an URL:

    foreach(var package in packages)
    {
        var url = string.Format(@"https://www.nuget.org/api/v2/package/{0}/{1}",
            package.Id, package.Version);
        var text = string.Format(@"{0} (v{1})", package.Id, package.Version);
        new Hyperlinq(url, text).Dump();
    }
    

    Which give us a nice clickable list of packages starting with a string directly in LINQPad output :

    Links in output

    Pricing

    Regarding the price for free you get the basic UI, the Pro versions is at 40$ (Autocompletion) and the Premium (Direct SQL table editing + NuGet) is at 75$.

    The price is small (and often discounted) but as Roslyn is Open Source and make creating IDE text editors with auto-completion a lot easier I expect movement in this space especially for the Pro version features.

  • Blog engine changed to Jekyll"

    This weekend I wanted to blog about libgit2sharp but I realized that my setup for blogging wasn't really practical. I used dotclear but by insisting on markdown and refusing the graphical editor it was only an hindrance for me.

    In response I investigated the domain of the simple blogging platforms running only locally and generating html files. The one I ended up using is a ruby tool named Jekyll; the one that is used by GitHub for their GitHub pages.

    What is nice is that using a web-hook on my blog repository the simple fact of pushing a new markdown file or modifying a sass file will re-generate all my blog with the changes.

  • Using CommandBindings in MVVM

    One sad thing in WPF currently is that the CommandBindings used to attach an action to a RoutedCommand don't support the MVVM pattern : Their properties can't be easily bound to the DataContext, and they don't support attaching a command represented by an ICommand.

    One of the solutions have always been to use the command sink that Josh Smith provided in it's CodePlex article Using RoutedCommands with a ViewModel in WPF. But it's syntax force multiple changes on the view and also force the model to implements the Execute and CanExecute independently out of an ICommand and don't begin to support the CanExecuteChanged event.

    My solution was to roll my own classes, that you could find in a Gist on GitHub to use it instead of a CommandBindings block in your UIElement you declare an attached property called Mvvm.CommandBindings and use it with nearly the same syntax :

        <Window x:Class="BlackFox.SampleWindow"
                xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
                xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
                xmlns:f="clr-namespace:BlackFox"
                >
            <f:Mvvm.CommandBindings>
                <f:MvvmCommandBindingCollection>
                    <f:MvvmCommandBinding
                       Command="f:MyRoutedCommands.SomeRoutedCommand"
                         Target="{Binding MyCommandInViewModel}"
                         CanExecuteChangedSuggestRequery="True" />
                </f:MvvmCommandBindingCollection>
            </f:Mvvm.CommandBindings>
        </Window>
    

    The only differences are that there is a Target (ICommand) property instead of the methods and that you could optionally force a CommandManager.InvalidateRequerySuggested(); when the command's CanExecuteChanged event is raised.

  • Using svndumpfilter to extract a folder in it's own repository

    What i'll explain here is how to use the svndumpfilter program to extract a folder of an SVN repository in it's own repository and make it disappear from the history of the original one.

    What you'll need

    You will need direct access to the server and to warn all your users that they'll need to checkout again everything you will move (if you don't renumber the revisions otherwise all users will need to checkout again any part of the repository they use)

    Knowing what to filter

    You will need to include all the folders you want along with all the folders where something moved, for example if you want /some/dir but it was at some point in the history named /somedir and renamed later you need to include both in the filter.

    The same apply if only some of the files moved from there in this case you need to include each file that moved.

    You'll end up with a list of files and directories representing all the history of the files that will be extracted.

    Dump the repository

    The first step is to dump the repository as svndumpfilter only act on dumps and not on directories themselves

    svnadmin dump repo > repo.dump
    

    Filter the history and load it in a new repository

    To create a dump that contains only the change in our chosen directories the command is :

    svndumpfilter include --drop-empty-revs --renumber-revs /some/dir /somedir < repo.dump > repo-only-some-dir.dump
    

    The two options --drop-empty-revs and --renumber-revs are what you'll need in 99% of the cases but anyway you'll either want both or none :

    • Both will give you a clean new repository with new revisions starting at zero and no empty revisions. It's clean but if you had anywhere else (in the source code comments, in your issue tracker or in some commit message) something referencing "r3628" it will now be incorrect.
    • None will give you a chance to have the same revision numbers (Only if you don't need to create new directories otherwise all revision numbers will move, see next paragraph)

    Then you'll need to create a new repository

    mkdir newrepo
    svnadmin create newrepo
    

    If some of the directories you included had a parent that wasn't himself included (like our /some/dir directory) you need to create them back

    svn checkout "file:////path/to/newrepo/" newrepocheckout
    mkdir newrepocheckout/some
    svn add newrepocheckout/some
    svn commit newrepocheckout -m "Prepare to load filtered history"
    rm -R newrepocheckout
    

    Then finally load your filtered history :

    svnadmin load newrepo < repo-only-some-dir.dump
    

    Remove the files from the originating repository history

    It's simplier as you normally won't have to create any directory, just load the dump in place of the old repo.

    The same choice is offered to you regarding removing the now-empty revs from the history and renumbering the other revisions but in this case it's more logical to keep the revisions as it will allow for all users that hadn't checked out the removed directories to continue to work with their checkouts. Otherwise they'll need a clean new checkout.

    rm -R repo
    svndumpfilter include /some/dir /somedir < repo.dump > repo-without-some-dir.dump
    mkdir repo
    svnadmin create repo
    svnadmin load repo < repo-without-some-dir.dump
    
  • SSL Observatory how to load the db

    The SSL Observatory is a great project from the EFF to gather all SSL certificates available on HTTPS websites and publish them for analysis. For more details see the SSL Observatory website.

    The main problem if you don't want to analyse it using their pre-configured Amazon EC2 configuration is that the compressed .sql file is around 3 Go compressed using LZMA and 3 To uncompressed so you can't even un-compress the file and need to pipe directly from 7-Zip to mysql.

    To do it either on windows or linux :

    7z e -so "observatory-dec-2010.sql.lzma" f | mysql -u root --password=toto42sh observatory2
    

    Note: The 7z tools is in the p7zip package on a debian system.

  • Things to do after formating a Sheevaplug

    Once a SheevaPlug is formated with sheevaplug-installer the minimum to do is :

    Set the MAC address

    Connect to the serial console, enter U-Boot and set the MAC address :

    setevn ethaddr 00:50:43:01:D2:94
    saveenv
    reset
    

    Set timezone and hostname

    Login and type :

    # Set the time-zone
    dpkg-reconfigure tzdata
    #Set the host name
    echo plugfox.vbfox.local > /etc/hostname
    /etc/init.d/hostname.sh
    
  • Gmail "mark as unread" now works correctly !

    As explained in the Gmail blog, the really annoying bug of the "mark as unread" button is now fixed.

    Before if you had a new post in a really old thread with dozens of mails and marked it unread, all the messages in the thread were marked as unread, expanding them all ! It was really annoying as for long threads it meant scrolling a lot to find the last mail really unread message.

    Good job, but now a "mark all messages after this one as unread" option on each message seem needed.