FIX logs analyser

If you are engaged in development software for trading systems there can be a necessity of the FIX engine log analysis. There are a lot of reasons, the most simple - define how much there correspond parameters of the developed application of the validity. This mean: your database speaks you one supposedly warrants, openly have been sent so much so much, closed so much, and actually all can be differently. If the parameters received at the analysis of a database will differ from the parameters received at the analysis FIX engine logs - it is time to sound alarm. Someone says lies, and most likely this someone - your application:).

Log file of FIX engine usually represents a set of text lines. The choice of the tool for their analysis has come at once - certainly, Perl:). One of problems consists that supports FIX for Perl simply are not present. There is a plenty of the servers written in every possible languages (basically, it Java or C ++) among which it is possible to meet even Ruby and Python (I so understand, the craze), but hotly favourite by us Perl is not present. All right, not a problem. We can write analysis of text data even if there are no working on Perl engines.

There was a problem of analysis of a file for trades with one well known forex broker. The basic application from our party has been written on Java with use of QuickFIX/J engine. Obtained trades data are stored in the separate file which growing quickly enough. At the moment of start of a test script the size of log file exceeded the 1.5 GB. The script run on all lines, choosing all necessary types of the message and analyzing conditions of orders has been developed really quickly. The result (for a day) is printed the report of a sales volume with format usage.

The script is not ideal in view of a hardcoded binding of variables and there is a direct dependence from our broker (they have a little expanded standard protocol with the own types of variables) but the general idea is more or less clear.

In any case the script has made the work and has made it quickly. I think that the choice Perl as the tool of the analysis was more than is justified in view of speed of development and the general code understanding. Conclusions - I was very happy.

Source code

#!/usr/bin/perl -w
use strict;
use Order;
my $log_file = $ARGV[0];
my %data;
open (LOG, $log_file) or die "$!\n";
while() {
    $_ =~ /35=(\w\D)/;
    if($1 eq "AP") {
        #We are looking for PositionReport message only
        process_position_report();
    }
}
print_volume_report();
sub process_position_report {
    my @tags = split(/\x01/);
    my %message;
    foreach (@tags) {
        my ($key, $value) = split(/=/);
        $message{$key} = $value;
    }
    #Get transaction date
    my $date = (split(/-/,$message{60}))[0];
    my $symbols = $data{$date};
    unless(defined($symbols)){
        $symbols = {};
        $data{$date} = $symbols;
    }
    #Get Symbol name
    my $symbol = $message{55};
    my $orders = $symbols->{$symbol};
    unless(defined($orders)) {
        $orders = {};
        $symbols->{$symbol} = $orders;
    }
    #Get ClOrdID
    my $order_id = $message{11};
    #Get FXCMPosID
    my $external_id = $message{9041};
    my ($size, $side);
    #Here we get Size and Side of Position
    if(defined($message{705})) {
        $size = $message{705};
        $side = "S";
    }else{
        $size = $message{704};
        $side = "B";
    }
    my $order = $orders->{$external_id};
    unless(defined($order)) {
        $order = Order->new();
        $order->id($order_id);
        $order->symbol($symbol);
        $order->open_date($message{60});
        $order->external_id($external_id);
        $order->size($size);
        $order->side($side);
        $orders->{$external_id} = $order;
    }else{
        if(defined($message{9052})) {
            $order->size(2*$order->size);
        }
    }
}
sub print_volume_report {
    my %volume_data;
    my ($symbol, $symbol_volume);
    my $total_volume = 0;
format =
    @<<<<<<<< @>>>>>>>>>>
    $symbol, $symbol_volume
.
    foreach(keys %data) {
        my $volume_symbols = {};
        $volume_data{$_} = $volume_symbols;
        my $symbols = $data{$_};
        foreach my $symbol (keys (%$symbols)) {
            my $volume = 0;
            my $orders = $symbols->{$symbol};
            foreach(keys(%$orders)) {
                $volume+=$orders->{$_}->size;
            }
            $volume_symbols->{$symbol} = $volume;
        }
    }
    foreach(keys %volume_data) {
        my $day_volume = 0;
        print $_,"\n";
        my $symbols = $volume_data{$_};
        foreach(keys(%$symbols)) {
            $symbol = $_;
            $day_volume+=$symbols->{$_};
            $symbol_volume = format_size($symbols->{$_});
            write;
        }
        #Print footer. Day volume
        $symbol = "Total";
        $symbol_volume = format_size($day_volume);
        write;
    }
}
sub format_size {
    my $size = shift;
    if($size < 999999) {
        return ($size/1000)."K";
    }elsif($size < 999999999) {
        return ($size / 1000000)."M";
    }else{
        return ($size / 1000000000)."B";
    }
}

package Order;
use strict;
use base 'Class::Accessor::Fast';
sub new {
    my $class = shift;
    my $self = bless {}, $class;
    return $self;
}
__PACKAGE__->mk_accessors(qw/
    id           symbol      side        size
    open_date    open_price  close_date  close_price
    type         external_id
/);
1;

Kiev
August 16, 2008
Serg V. Gulko

Trading foreign exchange on margin carries a high level of risk, and may not be suitable for all investors. The high degree of leverage can work against you as well as for you.
Before deciding to invest in foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with foreign exchange trading, and seek advice from an independent financial advisor if you have any doubts.