Eigenstate : Retrospective: 2016

Retrospective 2017

The year sucked. But work got done.

Retrospective 2016

2016 was a good year for Myrddin, with 574 commits to the compiler and standard libraries, hundreds more commits to other Myrddin related repositories and projects, several contributors and pull requests, ports to new operating systems and a number of interesting projects using Myrddin.

The summary, for those who don't care to read the whole page:

This year, the compilers and libraries seem to have matured enough that it's possible, even likely, to write programs without hitting any serious issues, which is no small feat for mostly one person reinventing most of the world. There are still known holes, deficiencies, and bugs, but they feel like they're small enough to tackle, especially with the fresh blood contributing and testing.

Andrew Chambers has written a few projects in Myrddin, the most impressive of which is his C compiler, qc. On top of being small and cleanly written, it's almost useful. It's, capable of natively using system headers on my FreeBSD box to compile nontrivial amounts of C code, using the QBE backend to generate optimized code. It lives at https://github.com/andrewchambers/qc

Quentin Carbonneaux has been valuable discussing ideas and laguage direction. In addition, he's helped helped discover and debug a number of issues in the type system. Additionally, it's likely that when the Myrddin compiler gets self hosted, it will be on his backend, qbe

New Myrddin Code

I realized I have nearly 200,000 lines of Myrddin sitting on my hard drive. Lots of it is duplicated, autogenerated, like my experiments with XCB bindings, or a result of failed experiments. Still, there is a great deal of code that exists.

When it comes to tools and code written in Myrddin, there has been a good deal of work done looking inwards, writing Myrddin to support writing more Myrddin. There are a number of tools to do things like dump memory statistics, which are only useful for Myrddin.

However, other generally useful tools have been written, including contbuild. Contbuild is a continuous builder which is running on the Myrddin website. Other neat tools include a regex debugger, useful for learning why regexes either match or fail to match a pattern. And as the language has matured, I've found myself writing small utilities in it, instead of pulling out either C or Python.

And while it is not yet ready to be released, there is work on a parser generator that fills ne needs for Lex and Yacc. This parser generator will be designed to generate output for more common languages, hopefully becoming a competent replacement for Lex and Yacc for Myrddin, C, C++, and whatever other languages people decide to add support for.

Libthread has landed this year. The library itself is very basic, providing only thread spawning and mutexes as things stand, but the basics are in place, and the way forward to provide a solid implementation on multiple platforms is clear. Of course, the higher level interfaces for threading are still somewhat up in the air, pending discussion.

use thread
const threadcount = {
    var tid, threads

    threads = [][:]
    for var i = 0; i < Nherd; i++
        tid = thread.spawn({
            std.put("nthreads: {}\n", threads.len)
        })
        std.slpush(&threads, tid)
    ;;
}

Right now, it works well on FreeBSD and Linux, and acceptably on Plan 9. Doing a good job on OSX and OpenBSD is a work in progress.

Libcrypto has gathered more functionality, including a secure RNG based off of OpenBSD arc4random, and a constant time implementation of AES. Hopefully over the next year, it will grow to become a viable base for a complete TLS implementation in Myrddin.

use crypto
const encrypt = {key, iv, data, result
    var ctx
    crypto.chacha20keysetup(&ctx, key)
    crypto.chacha20keysetup(&ctx, iv)
    crypto.chacha20encrypt(&ctx, data, result)
}

Libescfmt exists, and can handle escaping text for a number of different outputs. It's rather nifty, because it actually has no API for formatting, but instead uses libstd hooks for formatting.

const escape = {
    std.put("shell-escaped text: {}", escfmt.sh("foo & bar"))
    std.put("url-escaped text: {}", escfmt.url("foo & bar"))
}

Libmparse was written, and can parse Myrddin source. There are a number of incomplete parts, most notably the handling of usefiles and of type inference, but it's a good base for dealing with Myrddin source code.

Libjson has not been shipped with Myrddin yet, but it exists on the internet, works, and has over 300 tests to verify that it catches all sorts of edge cases.

const json = {
    var text, json

    text = std.try(std.slurp("test.json"))
    json = std.try(json.parse(text))
    std.put("{}\n", json)
}

Libinifile has been added, to allow for simple configuration. Ini files are a rather restricitve format, and parsing them is very simple, but because they are so restrictive, they are easy to understand and use.

Libhttp now works, at least on the client side. It could stand to be cleaned up and tested, and a server side implementation should be added. Once those are done, it will also land in the standard set of libraries shipped with Myrddin.

Libavl was written by Quentin, implementing AVL trees. The code there was contributed earlier this year, and caught a number of bugs in the type checking code, because of the specific nature of the recursion in the type checker.

Libstd has covered most of the glaring holes that were present at the start of the year, and presents nearly all the functions I find myself wanting as I write programs.

Language Changes

The biggest change to the language is that the changes to the language weren't big. Myrddin has actually stabilized quite a bit. There are pending changes to resource management. Additionally, there are a few restrictions around trait scoping that I would like to lift. However, I am largely happy with how things have turned out, and don't find myself wanting to make many changes.

The most impactful change this year to how code is written comes through pointer chasing pattern matching. You can now use the address operator in pattern matches to match what is pointed to by a pointer:

match intptr
| &123: std.put("the value pointed at by intptr is 123\n")
| _:    std.put("the value pointed at is not 123\n")
;;

In this case, the change is pretty pointless. However, if you want to match a complex data structure, for example, when tiling a syntax tree, this kind of chasing into data structures is extremely valuable.

Traits have also been made more powerful, allowing implementations over generics. When a trait is implemented over a generic, the most specific implementation is selected. This allows something that allows the most intereesting attributes of sfinae -- specialization of implementations for specific types -- without the clusterfuck of metaprogramming that traditionally comes with it.

Looking Forward

The next year is looking bright. The major goals coming up include:

Myrddin is nearly due for a release. Multiple users have been hacking on and with it, and bugs are rare enough that the language is usable. The language isn't stable, but it's settled down. And there are enough systems that many interested developers can use it.

Of course, right now there is very little documentation. The API documentation on this site is fine, but the language spec is not complete and not up to date. There is no tutorial, and there are few resources for learning the language and tools. For a release, that is going to have to change.

Once that's out of the way, I'm going to split my time between improving the libraries and weaning the implementation off of the GNU toolchain and the C language.

There's already a good deal of work towards self-hosting, with a parser library well underway. All that's left is to build a compiler on top of it. This will probably be either using QBE, or implementing a new SSA backend.

A new toolchain has also been started in collaboration with k0ga, author of the Suckless C compiler. The goal is that we will have compatible toolchains that are portable across platforms, and fit our needs better than the rather clunky and non-portable native toolchains. With multiple people working on it, this should hopefully go quickly.

When it comes to libraries, there are a number of things that still need to be worked on. First is improving the libraries used for web technologies. HTTP needs to be finished. TLS also needs to be implemented. Improving threading and async I/O come next.

There are also other libraries that end up getting used all over the place, like floating point math. Right now, I have no implementation of things like sqrt() or sin(), and that needs to get fixed.

I also plan to finish off a number of other tools that I have had on my back burner for ages. Getting a releasable version of the Hairless parser generator would be fantastic, for example.

Finally, Myrddin should be used for a number of tools that stand on their own, instead of just tools used to support more Myrddin development.

Overall, it's been a fun year of work, and I'm looking forward to using my own tools heavily going forward!