Binary Message Encoding

Learned a couple of really cool pieces of syntax since yesterday and it's all about Binaries. Developers can directly manipulate strings of binary information in a natural way in Erlang. Say you have a server somewhere that is written in C++. In my case I used Qt as my socket provider. The server packs in code like this:

union CharToInt 
{
    quint32 number;
    char buffer[ sizeof(quint32) ];
};

void packStringMessage(QString const& str, QTcpSocket& socket)
{
    CharToInt c;
    c.number = str.size();
    socket.write(c.buffer, sizeof(quint32));
    socket.write(str.toLocal8Bit()); //< Simple char-array.
}

This code is simple. It encodes a size of the string followed by the string itself. In Erlang you can simply do the following (once you have the message):

<<Size:32/little, Payload/binary>> = Message.

It will store the size of the message inside of Size and the rest of the binary data inside of Payload! What are the arguments all about? Well let's dissect:

<<Size:32/little, Payload/binary>> = Message.

 Binaries are inside of << and >>. Whatever is inside is the makeup of the binary.

<<Size:32/little, Payload/binary>> = Message.

Notice that we can do the same pattern matching strategy as lists/function calls. I think this is the defining feature of Erlang.

<<Size:32/little, Payload/binary>> = Message.

 The Size token is an identifier stating what variable the matching binary will be put inside of. The :32 specifies how many bits we are using. I'm taking a lot of liberties with this example; assuming quint32 fits in 4 bytes and that we are storing in little-endian notation. That's what the /little states to Erlang. The data that Size corresponds to should be 32 bits and in little-endian notation! Really convenient.

<<Size:32/little, Payload/binary>> = Message.

The final part to this is that the end of the binary (everything else) will be stored in Payload and should be binary data. I was surprised at how easy this was to decode binary messages in Erlang.

Redis

Taking a break from Erlang momentarily to install Redis. Redis is a database that specializes in storing key-value pairs.

Getting Redis

1) Go to the Redis downloads page and download the latest tarball source. http://redis.io/download I'm using version 2.8.6, which is the latest stable version as of today's date.

2) Unpack Redis:

> gunzip redis-2.8.6.tar.gz
> tar -xf redis-2.8.6.tar

3) Step into the redis directory and build the source:

> cd redis-2.8.6
> make
> sudo make install

At this point I'd like to give props to the Redis developers. To have a Makefile that "just works" is phenomenal! So far I'm really liking Redis- if only for the ease of building/installation.

Running Redis

I should also note at this point- similar to how yaws requires an instance to be running either as daemon or interactive, redis is the same way. To run the server in an interactive mode:

1) Open a Terminal.

2) Type:

> redis-server

You should see some decorative text. If you haven't done this step you may get an error message such as:

Could not connect to Redis at 127.0.0.1:6379: Connection refused

Not good. So to avoid this, run redis-server and be happy.

Erlang Tie-In

Introducing eredis, by Github user wooga. This is some cool stuff- you can use the instructions at https://github.com/wooga/eredis to connect to the redis server from your Erlang application! The instructions are good so I won't repeat them here. Something interesting to note is that the function eredis:q takes two parameters: a connection to the redis database and a list. The list is cool because it's the string components of the query. If you are going to set some data:

eredis:q( Connection, [ "SET", "mykey", "Hello World" ] ).

It doesn't get much simpler. The return from this function is a tuple where the first element should be "ok" or whatever the resulting status was. The second argument will be whatever the values returned are.

Conclusion

Being unsure about databases I really like redis. It's very simple- just key/value. Connecting to it couldn't be easier. Running it couldn't be easier. I'm thinking that the purpose of this post is to try and get the word out about this. If I run into any problems later I'll come back to do an addendum.

Makefile

Introduction
The way to structure projects in Erlang is a mystery to me. In C++, it's simple because everything names itself in the projects:

/project
    include/
        project /
    src/
    test/
    resources/
    views/

Then in the root directory I normally place my qmake or premake4 files. This seems like second nature. After building we'd also end up with some other directories to hold binaries, libraries, and object files.

In Erlang we have the notion of macro/definition files (hrl's); we also have source code (erl's). The compiled output for Erlang is a beam file. How does everything normally get structured? To better answer this question I thought I'd examine the Erlang makefile.

Getting It

I grabbed a version from git:

> git clone https://github.com/extend/erlang.mk.git

This gives you erlang.mk. Have this handy to copy into your experimental directory.

Trying it Out

Follow the steps below to make a small test:

> mkdir experiment
> cd experiment
> cp erlang.mk ./ # Or wherever erlang.mk exists.
> vim Makefile # Or your editor of choice.

Inside of Makefile, just add:

include erlang.mk

And save. Back in the terminal try this:

> make

This won't work because we don't have any sources set up! It will complain with:

find: `src': No such file or directory
find: `src': No such file or directory
find: `src': No such file or directory
find: `src': No such file or directory
 APP    experiment.app.src
cat: src/experiment.app.src: No such file or directory

This is because we need to make a directory called src and put some Erlang sources inside of it:

> mkdir src
> cd src
> vim test1.erl

Put a test program inside of test1.erl, could be something simple like:

-module(test1).
-export([blah/0]).

blah() -> "Hello".

After saving, step down a directory and try to make again:

> cd ..
> make

This time it will create a directory called "ebin"; this is where all of the beam files end up. Some additional notes on structuring Erlang programs can be found here:

http://www.erlang.org/doc/design_principles/applications.html#7.4

The hrl files can go into an include directory.

Dependencies

We have our source code but we need to use other people's libraries/projects in our own. The Erlang Makefile is like a dream come true for this purpose: it can automatically pull down your required dependencies directly from git. I suspect it can do more than just that but for now we'll end with that.

In the line before include erlang.mk:

DEPS = eredis
dep_eredis = https://github.com/wooga/eredis.git
include erlang.mk

Now try building. Pretty impressive isn't it? The format here is to list your dependencies after DEPS, then for each entry you add a dep_ENTRY = followed by the location of the target library. The Makefile then automates building everything for you. I can't emphasize how cool this is, although most Erlang developers probably already know it.

Records and HRL Files

Looking online I noticed that there is a lack of good "clear" examples on how to define records inside of hrl files (and use them). Seems like all of the examples have a slight twist. Hopefully these instructions help the beginners out.

The hrl files are used for defining macros and records. I don't know anything about macros [yet]; records are basically fancy tuples where you can "name" values. I'm putting together some page properties for a website I want to build and I want to put all of the display information into one place. I could use tuples and do the following:

-module(page_properties).
-export([all_props/0]).

all_props() -> 
    [
         { bgcolor, "#444444" },
         { bgimage, "someimage.jpg" },
         { h1, "<h1>" }
   ].

Then I would use the all_props() definition to obtain information about what the background color is, or how to define a heading region.This is all well and good- however it forces the users to know the exact tuple layout or it forces us to create accessor functions. Here's an example of how I've done accessors to grab values from tuples:

-module(page_properties).
-export([all_props/0, get_property/1]).

all_props() -> 
    [
         { bgcolor, "#444444" },
         { bgimage, "someimage.jpg" },
         { h1, "<h1>" }
   ].

get_property(Name) -> 

    get_property( all_props(), Name).

get_property( [H | T], Name) ->

    { CurrentName, Value } = H,

    if 

        CurrentName =:= Name -> Value,

        true -> get_property( [], Name )

    end;

get_property( [], Name ) -> 0.

Whew, that's a lot of code to try and obtain property values. All that is left is to access the correct property (from inside the shell):

page_properties:get_property( bgcolor ).

This code isn't too bad- but why would you implement this mechanism when Erlang already has a facility built-in that could do it better? That's where records come in. First off, let's create an "hrl" file to define our record. This is a new example, disregard everything above!

Inside of page_properties.hrl:

-record( page_props,

     {

         bgcolor = "#444444",

         bgimage = "background.jpg"

    }).

Now inside of html_body.erl:

-module(html_body).

-include("page_properties.hrl").

-export( [get_bgcolor/0] ).

get_bgcolor() -> #page_props{ bgcolor = Value } = #page_props, Value.

That's all there is to it! Alternately you can define get_bgcolor as:

get_bgcolor() -> A = #page_props{}, A#page_props.bgcolor.

Not sure which one is preferable. If you are trying to get more than one value back from your record, I'm assuming that the pattern-matching mechanism works best. 

Please note the order in which -module, -include, and -export occur. The -include must be done after -module, otherwise you'll get an error stating that a module hasn't been defined! Notice as well that the include takes a string argument to an actual hrl file- this seems different than every other directive I've encountered thus far. You can actually use absolute paths and environment variables here! Pretty cool stuff.

The benefit to defining your code in a record inside of an hrl file is as follows:

- One place for all definitions that is clear and concise.

- No need for memorizing the layout of a tuple.

- No need for fancy code to traverse a tuple to extract values.

        

YAWS + External Modules

One of the first questions I had when setting up a YAWS page was, "How do I add more Erlang modules to the run?". The answer was more involved than I thought! A basic YAWS webpage (the simplest) is:

<html>
<body>
    <erl>
        out(A) -> "Hello World".
    </erl>
</body>
</html>

This is a simple webpage that prints out Hello World. What if you had already written some really cool erlang code that was outside of yaws? This is the code I was working on:

-module(page_properties).
-export([get_background_color/0]).

get_background_color() -> "#444444".

If you are typing this in, make sure you put it into a file called "page_properties.erl". The name of the file must match the module name! To build this example go into the erlang shell from Linux:

> erl

Then inside the shell, use the c function to compile. The result of compiling will be a "beam" file in the same directory. The beam file is the compiled output and it's what can be run.

> c(page_properties).

So c(page_properties) will look for page_properties.erl (because the name of the module matches the file name) and compile it and bring it into the shell session. So now I can do this:

> page_properties:get_background_color().

And it should return the background color you provided. Back to YAWS now- how do we get this code to be available inside of YAWS? Inside of YAWS there is only so much you can fit inside of the out(A) function. To load external modules into the yaws session open two terminal windows. Inside one, run the YAWS server interactively:

> sudo yaws -i

In the second terminal, navigate to the same directory that has the beam file you just built (or want to include). Then do the following:

yaws -- load page_properties

This will load page_properties into the running server! Surprisingly this was all it took. This is cool because if you have a running server and you don't want to take it down because it's live you can easily load in new code without skipping a beat! Now the yaws webpage can use the loaded module:

<html>
<body>
    <erl>
        out(A) -> Color = page_properties:get_background_color(),
                  string:join( [ "The color is: "
                               , Color ], "" ).
    </erl>
</body>
</html>

The Color is first set to the string provided by our external module, then we use string:join to concatenate the two strings together. You might ask why string:join can be used, yet other external modules have to be loaded manually- this is because some modules are "sticky", meaning they are always loaded during every run. 

YAWS Setup

Introduction

YAWS, short for "Yet Another Web Server" is completely written in Erlang. It's prided on being able to handle more traffic than Apache [1]. It's the technology that should have been used by the healthcare website (just kidding). 

This guide will cover from start to finish the steps necessary to create a YAWS based server. This will be thorough as I have no experience with Erlang or web-technologies. If you are new to Erlang or to websites, this will be the guide for you. If you are an advanced user this could be used as a sleep-aid. This guide is setup into different sections. The first is obviously going to be prerequisites.

Prerequisites (What do I need?)

I'm going to set up YAWS on a CrunchBang distro (it's the same for Debian/Ubuntu). It's a 64-bit virtual machine running a Bridged network adapter. The nice thing about running in a Virtual Machine is that anyone can do this! Go grab an ISO for Debian/Ubuntu/LinuxMint/CrunchBang or whatever your flavor of the month is and install it to a virtual machine. I use VirtualBox because it's free and easy to use. 

When setting up your virtual machine make sure to set the network adapter to Bridged. I've heard from colleagues that the NAT adapter has issues. The other reason you want Bridged is so that the network can communicate "more directly" to the virtual machine without having to go through the host machine.

From the linux command line:

> sudo apt-get install erlang

Erlang is needed because it's the base language that YAWS is written in.

Getting YAWS

The first step is to use wget to obtain the source code for yaws. I don't recommend using apt-get to install yaws as the version in the repo seems to have problems. It is recommended to go right to the yaws website to obtain. Open this link in a web browser:

http://yaws.hyber.org/download

This will take you to a list of downloads. Find the most current download; currently it is yaws-1.98.tar.gz. So from the command line do the following:

> wget  http://yaws.hyber.org/download/yaws-1.98.tar.gz

This will download it to the current directory you are in. Now unpack:

> gunzip yaws-1.98.tar.gz
> tar -xf yaws-1.98.tar

All done. Source code is ready to go. Step into the yaws-1.98 directory:

> cd yaws-1.98

Try to configure and build the project:

> ./configure
> make

The first thing I saw was an error two files in stating:

epam.c:2:22: fatal error: pam_appl.h: No such file or directory.

Looks like I'm missing some dependencies! If you get this error, just do the following:

> sudo apt-get install libpam0g-dev

After that my build was successful! The next step is to install yaws:

> sudo make install

This will put yaws into the appropriate place on your system and also it will put together some default configuration files for you. To test out yaws do the following:

> sudo yaws -i 

The -i flag on yaws will put you into interactive mode. If you then point your web browser at 127.0.0.1, you'll see a nice little YAWS page!

First Page

First run the yaws web server: 

 

> sudo yaws -i

 

Pay close attention to the output printed to the console. One of the lines should look like:

=INFO REPORT==== 21-Feb-2014::11:09:21 ===
Yaws: Listening to 0.0.0.0:80 for <2> virtual servers:
 - http://servername under /usr/local/var/yaws/www
 - http://localhost under /tmp

I've highlighted the text of interest. The gold line, 0.0.0.0:80 is stating that the server is listening on address 0.0.0.0 with port 80. When we are listening to address 0.0.0.0, it means any IPv4 address. If you did a netstat -l in Linux, you'll see all of your other servers running and the address they are bound to.

The blue line is what is more interesting. The blue part shows where the web server is running from. Take this value and navigate to that directory:

> sudo su   # Change to root.
> cd /usr/local/var/yaws/www
> ls

You should see a file called index.yaws. This is the file that is first loaded when you point your browser at the server's main site. Try this:

> mv index.yaws index.yaws-backup
> vim index.yaws

Type in the following program to get started:

<html>
    <body>

        <erl>
            out(A) -> { html, "Hello World!" }.
        </erl>

    </body>
</html>

It's that simple! When you run your server:

> sudo yaws -i

And point your web browser at your server, you should see the "Hello World!" message!