Knowledge Base

AmiBroker Formula Language (AFL) thanks to its array processing model is able to run at the same speed as code written in assembler (i.e. machine code). The following article explains how.

AFL runs with native assembly speed when using array operations.
A simple array multiplication like this:

X = Close  * H; // array multiplication (each array element is multiplied)


gets compiled by AmiBroker to just 8 assembly instructions:

loop: mov         edx,dword ptr [esp+58h]
      inc         esi                       ; increase counters
      add         eax,4
      cmp         esi,edi
      fld         dword ptr [edx+esi*4-4]   ; get element of close array
      fmul        dword ptr [eax+ecx-4]     ; multiply by element of high array
      fstp        dword ptr [eax-4]         ; store result
      jl          loop                      ; continue until all elements are processed

As you can see there are three 4 byte memory accesses per loop iteration (2 reads each 4 bytes long and 1 write 4 byte long)

With such tight loop, single processor core running an AFL formula is able to saturate memory bandwidth in majority of most common operations/functions if total array sizes used in given formula exceedes DATA cache size.

On my (2 year old) 2GHz Athlon x2 64 single iteration of this loop takes 6 nanoseconds (see benchmark code below). 6 nanoseconds to process one bar of data, or 166 million bars per second. So, during 6 nanoseconds we have 8 byte reads and 4 byte store. Thats (8/(6e-9)) bytes per second = 1333 MB per second read and 667 MB per second write simultaneously i.e. 2GB/sec combined !

Now if you look at memory benchmarks you will see that 2GB/s is the limit of system memory speed on Athlon x64 (DDR2 dual channel)
And that’s considering the fact that Athlon has superior-to-intel on-die integrated memory controller (hypertransfer)

// benchmark code
// for accurrate results run it on LARGE arrays -
// intraday database, 1-minute interval, 50K bars or more)
GetPerformanceCounter(1); 
for(k = 0; k < 1000; k++ )  X = C * H; 
"Time per single iteration [s]="+1e-3*GetPerformanceCounter()/(1000*BarCount); 


Only really complex operations that use *lots* of FPU (floating point) cycles such as trigonometric (sin/cos/tan) functions are slow enough for the memory to keep up.

Filed by Tomasz Janeczko at 2:20 pm under AFL
No Comments

That is the common question we receive through support channel, so even though it is explained in the LICENSE.TXT file that you have in your AmiBroker folder, let us make some facts straight.

First take a look what the license says:

“REGISTERED VERSION:
One registered copy of the program may either be used by a single person who uses the software personally on one or more computers, or installed on a single workstation used non-simultaneously by multiple people, but not both.
You may not disclose registration key(s) to non-registered users.
Storage/Network Use. You may also store or install a copy of the Software on a storage device, such as a network server, used only to install or run the Software on your other computers over an internal network; however, you must acquire and dedicate a license for each separate computer on which the Software is run from the storage device. A license for the Software may not be shared or used concurrently on different computers.”

So the answer to most common question: “Can I install and use it on both desktop and laptop?” is YES, you can.

It is however NOT allowed to use single license to RUN SIMULTANEOUSLY on MULTIPLE machines connected via network, like for example running optimization on many machines in parallel. For that you must purchase license for every machine you are using simultaneously.

Filed by Tomasz Janeczko at 12:21 pm under Installation
No Comments

QuickAFL(tm) is a feature that allows faster AFL calculation under certain conditions. Initially (since 2003) it was available for indicators only, as of version 5.14+ it is available in Automatic Analysis too.

Initially the idea was to allow faster chart redraws through calculating AFL formula only for that part which is visible on the chart. In a similar manner, automatic analysis window can use subset of available quotations to calculate AFL, if selected “range” parameter is less than “All quotations”.

So, in short QuickAFL works so it calculates only part of the array that is currently visible (indicator) or within selected range (Automatic Analysis).

Your formulas, under QuickAFL, may or may NOT use all data bars available, but only visible (or “in-range”) bars (plus some extra to ensure calculation of used indicators), so when you are using Close[ 0 ] it represents not first bar in entire data set but first bar in array currently used (which is just a bit longer than visible, or ‘in-range’ area).

The QuickAFL is designed to be transparent, i.e. do not require any changes to the formulas you are using. To achieve this goal, AmiBroker in the first execution of given formula “learns” how many bars are really needed to calculate it correctly.

To find out the number of bars required to calculate formula AmiBroker internally uses two variables ‘backward ref’ and ‘forward ref’.

‘backward ref’ describes how many PREVIOUS bars are needed to calculate the value for today, and ‘forward ref’ tells how many FUTURE bars are needed to calculate value for today.

If these numbers are known, during execution of given formula AmiBroker takes FIRST visible (or in-range) bar and subtracts ‘backward ref” and takes LAST visible (or in-range) bar and adds ‘forward ref’ to calculate first and last bar needed for calculation of the formula.

Now, how does AmiBroker know a correct “backward ref” and “forward ref” for the entire formula?
Well, every AmiBroker’s built-in function is written so that it knows its own requirements and adds them to global “backward ref” and “forward ref” each time given function is called from your formula.

The whole process starts with setting initial BackwardRef to 30 and ForwardRef to zero. These initial values are used to give “safety margin” for simple loops/scripts.

Next, when parser scans the formula like this:

Buy = C > Ref ( MA( C, 40 ), -1 );


it analyses it and “sees” the MA with parameter 40. It knows that simple moving average of period 40 requires 40 past bars and zero future bars to calculate correctly so it does the following (all internally):

BackwardRef = BackwardRef + 40;
ForwardRef = ForwardRef + 0;


So now, the value of BackwardRef will be 70 (40+30(initial)), and ForwardRef will be zero.

Next the parser sees Ref( .., -1 );

It knows that Ref with shift of -1 requires 1 past bar and zero future bars so it “accumulates” requirements this way:

BackwardRef = BackwardRef + 1;
ForwardRef = ForwardRef + 0;


So it ends up with:

BackwardRef = 71;
ForwardRef = 0;


The BackwardRef and ForwardRef numbers are displayed by AFL Editor’s Tools->Check and Profile as well as on charts when “Display chart timing” is selected in the preferences.

If you use Check and Profile tool, it will tell you that the formula

Buy = C > Ref ( MA( C, 40 ), -1 );


requires 71 past bars and 0 future bars.

You can modify it to

Buy = C > Ref ( MA( C, 50 ), -2 );


and it will tell you that it requires 82 past bars (30+50+2) and zero future bars.

If you modify it to

Buy = C > Ref ( MA( C, 50 ), 1 );


It will tell you that it needs 80 past bars (30+50) and ONE future bar (from Ref).

Thanks to that your formula will use 80 bars prior to first visible (or in-range) bar leading to correct calculation result, while improving the speed of execution by not using bars preceding required ones.

IMPORTANT NOTES

It is very important to understand, that the above estimate requirements while fairly conservative,
and working fine in majority of cases, may NOT give you identical results with QuickAFL enabled, if your formulas use:
a) JScript/VBScript scripting
b) for/while/do loops using more than 30 past bars
c) any functions from external indicator DLLs
d) certain functions that use recursive calculation such as very long exponential averages or TimeFrame functions with much higher intervals than base interval

In these cases, you may need to use SetBarsRequired() function to set initial requirements to value higher than default 30. For example, by placing

SetBarsRequired( 1000, 0 );


at the TOP of your formula you are telling AmiBroker to add 1000 bars PRIOR to first visible (or in-range) bar to ensure more data to stabilise indicators.

You can also effectively turn OFF QuickAFL by adding:

SetBarsRequired( sbrAll, sbrAll );


at the top of your formula. It tells AmiBroker to use ALL bars all the time.

It is also worth noting that certain functions like cumulative sum (Cum()) by default request ALL past bars to guarantee the same results when QuickAFL is enabled. But when using such a function, you may or may NOT want to use all bars. So SetBarsRequired() gives you also ability to DECREASE the requirements of formula. This is done by placing SetBarsRequired at the END of your formula, as any call to SetBarsRequired effectively overwrites previously calculated estimate. So
if you write

x = Cum( 1 );
SetBarsRequired( 1000, 0 ); // use 1000 past bars DESPITE using Cum()


You may force AmiBroker to use only 1000 bars prior first visible even though Cum() by itself would require all bars.

It is also worth noting that when QuickAFL is used, BarIndex() function does NOT represent elements of the AFL array, but rather the indexes of ENTIRE quotation array. With QuickAFL turned on, an AFL array is usually shorter than quotation array, as illustrated in this picture:

SPECIAL CASE: AddToComposite function

Since AddToComposite creates artificial stock data it is desirable that it works the same regardless of how many ‘visible’ bars there are or how many bars are needed by other parts of the formula.

For this reason internally AddToComposite does this:

SetBarsRequired( sbrAll, sbrAll );


which effectivelly means “use all available bars” for the formula. AddToComposite function simply tells the AFL engine to use all available bars (from the very first to the very last) regardless of how formula looks like. This is to ensure that AddToComposite updates ALL bars of the composite

The side-effect is that “Check And Profile” feature will see that it needs to reference future bars and display a warning even though this is false alert because AddToComposite itself has no impact on trading system at all.

Now why this shows only when flag atcFlagEnableInBacktest is on ??
It is simple: this is so because it means that AddToComposite is ACTIVE in BACKTEST.
http://www.amibroker.com/guide/afl/afl_view.php?name=ADDTOCOMPOSITE

Since “Check And Profile” uses “BACKTEST” state you get such result.

If atcFlagEnableInBacktest is not specified AddToComposite is not enabled in Backtest and hence does not affect calculation of BackwardRef and ForwardRef during “Check And Profile”.

BACKWARD COMPATIBILITY NOTES

a) QuickAFL is available in Automatic Analysis in version 5.14.0 or higher
b) sbrAll constant is available in Automatic Analysis in version 5.14.0 or higher. If you are using older versions you should use numeric constant of: 1000000 instead.

Filed by Tomasz Janeczko at 5:33 am under AFL, Custom Backtest, Explorations, Indicators, Systems
2 Comments

Recently on the AmiBroker mailing list some users expressed wish to have access to some of portfolio backtest metrics available in “historical” form (i.e. as date series, as opposed to scalars), so they can be plotted as an indicator.

Implementing such functionality is actually easy with existing tools and does not require any OLE scripts. Everything you need is small custom-backtester procedure that just reads built-in stats every bar and puts them into composite ticker.
In the accompanying indicator code all you need to do is simply use Foreign() function to access the historical metrics data generated during backtest.

The code below shows the BACKTEST formula with custom backtester part:

// Replace lines below with YOUR TRADING SYSTEM
EnableRotationalTrading(); 
PositionScore = 1/RSI(14); 
PositionSize = -25;
SetOption("WorstRankHeld", 7 );
SetOption("MaxOpenPositions", 4 ); 

In the code above, for illustration purposes, we are exporting UlcerIndex and Winners Percent metrics as data series. They are stored in composite tickers for easy retrieval from indicator level.
You can easily extend code to include ANY number of metrics you want.

Now in order to Plot metrics as indicators, use this simple formula:

PlotForeign("~~~UI_HISTORICAL", "UlcerIndex Historical", colorRed, styleLine );
PlotForeign("~~~WP_HISTORICAL", "Winners Percent", colorBlue, styleLine | styleOwnScale );

As you can see with one Foreign function call you can read the historical value of any metric generated by the backtester.

NOTE: when running backtest please setup a filter in AA that EXCLUDES composites (group 253) from backtest set.

Filed by Tomasz Janeczko at 4:22 am under AFL, Custom Backtest
No Comments

Recently I heard the suggestion to add a security symbol written in big letters in the chart background. Well, actually it is pretty simple to do using low-level gfx. Just add this code sniplet anywhere in your chart formula.

GfxSetOverlayMode(1);
GfxSelectFont("Tahoma", Status("pxheight")/2 );
GfxSetTextAlign( 6 );// center alignment
GfxSetTextColor( ColorRGB( 200, 200, 200 ) );
GfxSetBkMode(1); // transparent
GfxTextOut( Name(), Status("pxwidth")/2, Status("pxheight")/12 );


UPDATE: I have added transparent mode, so it works fine on non-white backgrounds too.

Filed by Tomasz Janeczko at 4:41 am under AFL, Indicators
No Comments

Recently released AmiBroker 5.05 BETA features the automatic Walk-Forward Optimization mode.

The automatic Walk forward optimization is a system design and validation technique in which you optimize the parameter values on a past segment of market data (”in-sample”), then test the system forward in time on data following the optimization segment (”out-of-sample”). You evaluate the system based on how well it performs on the test data (”out-of-sample”), not the data it was optimized on.

To use Walk-Forward optimization please follow these steps:

1. Goto Tools->Automatic Analysis
2. Click Settings button, then switch to “Walk-Forward tab”
3. Here you can see Walk forward settings for In-sample optimization, out-of-sample backtest
   “Start” and “End” dates mark initial period begin / end
   This period will be moved forward by “Step” until the “End” reaches the “Last” date.
   The “Start” date can move forward by “step” too, or can be anchored (constant) if “Anchored” check is on.
   If you mark “Use today” then “Last” date entered will be ignored and TODAY (current date) will be used instead

   By default an “EASY MODE” is selected which simplifies the process of setting up WF parameters.
   It assumes that:
   a) Out-of-sample segment immediatelly follows in-sample segment
   b) the length of out-of-sample segment equals to the walk-forward step

   Based on these two assumptions the “EASY” mode takes in-sample END date and sets
   out-of-sample START date to the following day. Then adds in-sample STEP and this becomes out-of-sample END date.
   In-sample and Out-of-sample step values are set to the same values.

   The “EASY” mode guarantees correctness of WF procedure settings.

   In the “ADVANCED” mode, the user has complete control over all values, to the extent that
   they may not constitute valid WF procedure.
   The interface allows to selectivelly disable in-sample and out-of-sample phases using checkboxes at top
   (for special things like runnign sequential backtests without optimization).

   All settings are immediatelly reflected in the PREVIEW list that shows all generated IS/OOS segments and their dates.

   The “Optimization target” field defines the optimization raport COLUMN NAME that
   will be used for sorting results and finding the BEST one. Any built-in column can be used
   (as appears in the optimization output), or you can use any custom metric that you define
   in custom backtester. The default is CAR/MDD, you can however select any other built-in metric from the combo.
   You can also TYPE-IN any custom metric that you have added via custom backtester interface.

4. Once you defined Walk-Forward settings, please go to Automatic Analysis and
5. press the dropdown ARROW on the Optimize button and select “Walk Forward Optimization”

This will run sequence of optimizaitons and backtest and the results will be displayed in the “Walk Forward” document that is open in the main application frame.
When optimization is running you can click “MINIMIZE” button on the Progress dialog to minimize it - this allows to see the Walk Forward output during the optimization steps.

IN-SAMPLE and OUT-OF-SAMPLE combined equity

Combined in-sample and out-sample equities are available by
~~~ISEQUITY and ~~~OSEQUITY composite tickers (consecutive periods of IS and OOS are concatenated and scaled to
maintain continuity of equity line - this approach assumes that you generally speaking are compounding profits)
To display IS and OOS equity you may use for example this:

PlotForeign("~~~ISEQUITY","In-Sample Equity", colorRed, styleLine); 
PlotForeign("~~~OSEQUITY","Out-Of-Sample Equity", colorGreen, styleLine); 
Title = "{{NAME}} - {{INTERVAL}} {{DATE}} {{VALUES}}"; 


Filed by Tomasz Janeczko at 7:54 am under AFL, Systems
3 Comments

The code below is an little bit more complex example of Low Level Graphics functions (see http://www.amibroker.com/guide/a_lowlevelgfx.html)

It allows to display three kinds of charts:

1. yearly/monthly profit table
2. yearly profit bar chart
3. average monthly profit bar chart

The type of chart is switchable from Parameters dialog.

It should be applied to ~~~EQUITY - portfolio equity symbol (so it only produces output if you run backtest before using it).

Figure 1. Profit chart in table mode

Figure 2. Profit chart in yearly mode

Figure 3. Profit chart in monthly mode

Filed by Tomasz Janeczko at 6:34 pm under AFL, Indicators
5 Comments

There are a couple of things you need to know about Yahoo Finance pages that AmiQuote uses to download “historical” and “current” quotes.

Current quotes are quotes for current day (or previous day if there is no trading session today). For example MSFT current quote page is here:
http://finance.yahoo.com/q?s=MSFT
AmiQuote uses rather “download data” link which is: http://download.finance.yahoo.com/d/quotes.csv?s=MSFT&f=sl1d1t1c1ohgv&e=.csv
But it is not relevant because both show same current quote.

Now there is a second source. Historical quotes are downloaded from Historical Prices page. For example MSFT historical page is here: http://finance.yahoo.com/q/hp?s=MSFT (again AmiQuote uses rather plain text link: http://ichart.finance.yahoo.com/table.csv?s=MSFT&d=7&e=4&f=2007&g=d&a=2&b=13&c=1986&ignore=.csv )

Why using two sources? That’s simple: current mode gives data during trading session, while historical is only updated many hours after markets close so both compliment each other. Current mode is also much faster as it downloads as many as 200 symbols at once, while historical must download one by one. So recommended usage is to use Yahoo Current mode everyday, and Yahoo Historical once a week.
Of course you may use historical everyday as well if you have time and fast internet connection.

It is important to understand that AmiQuote is just the downloader (like Internet Explorer) and it does nothing except downloading the data, so if you belive that there is a bad quote - it is not AmiQuote, but rather Yahoo problem. To verify always go to relevant page (see links above) and check the quote on Yahoo Finance site directly.

It may happen that quotes on Yahoo Current page and Yahoo Historical pages differ. It is so because Yahoo gets them from different data vendors. If this happens the only solution is to report data error to Yahoo.

There are however 2 things you need to know about importing of data:

1. AmiBroker by default imports split and dividend adjusted data (”Adj. Close” on Yahoo Historical page). For more information on how data are adjusted see Yahoo Help page at: http://help.yahoo.com/l/us/yahoo/finance/quotes/quote-12.html

2. AmiBroker by default imports volume in HUNDREDS of shares (so if volume is 183,200 shares, the volume chart will show 1832)

These things are adjustable, so if you don’t like them, you can change them.
The import process of historical quotes is controlled using aqh.format file that you will find inside “Formats” subfolder. By default it looks as follows (you can open it with Windows Notepad).

# AmiQuote historical quotes download format (.AQH extension)
$FORMAT Date_DMY,Open,High,Low,Close,Volume,AdjClose
$SKIPLINES 0
$BREAKONERR 0
$SEPARATOR ,
$DEBUG 1
$AUTOADD 1
$CONT 1
$GROUP 254
$VOLFACTOR 0.01


Lines marked with bold mark important areas.

$FORMAT line controls the import format. AdjClose field says that AmiBroker should use adjusted price. If you don’t want adjusted prices simply replace $FORMAT line with:

$FORMAT Date_DMY,Open,High,Low,Close,Volume
(note no AdjClose field)

$VOLFACTOR line controls the volume multiplier. If you want volume to be expressed in single shares instead of hundreds of shares replace $VOLFACTOR line with:

$VOLFACTOR 1

The same $VOLFACTOR change should be applied to aqd.format file that is responsible for importing data in Yahoo CURRENT mode (if you are using it).

Filed by Tomasz Janeczko at 2:19 pm under AmiQuote, Data
No Comments

In order to display indicators based on Advances/Declines first of all it’s necessary to calculate composities in the database:

1. Open Categories window using Symbol->Categories menu item.
2. Select base index for given market in Markets tab and Base indexes for - Composites combo.
   For example if you are following NYSE this can by ^DIJ (Dow Jones Average)
   (certain symbol must be marked as index in Symbol -> Information and must belong to the same market)
3. Choose Symbol ->Calculate composites  menu item to open the window shown below and mark:
   - Number of advancing/declining issues and
   - Apply to: all quotes, All markets
4. Click Calculate . From now on ADLine, AdvVolume() and TRIN indicators will be visible.

Q: Why does AB need “base index”?
A: Just because it may happen that not all stocks are quoted every businness day and AB needs must calculate number of advancing/declining issues per market. So it checks the “base index” quotations dates and tries to find corresponding quotes of all stocks belonging to that market to find out how many issues advanced, declined and not changed at all.

Q: What are “Volume for base index” and “Copy volume to all indexes” checkboxes for?
A: “Volume for base index” and “Copy volume to all indexes” are provided in case you DON’T have real volume data for index quotes. In that case AmiBroker can calculate volume for index as a sum of volumes of all stocks belonging to given market. First option assigns calculated volume only to “base index”, the second copies the volume figure to all indexes belonging to given market.

Filed by support at 7:01 am under Charting, Data
No Comments

There are many different ways to check for divergences. One of the simplest is to use Rate of change indicator and EXPLORATION feature of Automatic Analysis window:

- Analysis -> Formula Editor
- enter:
 
// 5 day rate of change of close
PriceUp = ROC( C, 5 ) > 0 ; 
// 5 day rate of change of MACD histogram
MacdUP = ROC( MACD() - Signal(), 5 ) > 0; 
BullishDiv = NOT PriceUP AND MACDUp;
BearishDiv = PriceUP AND NOT MACDUp;
Filter = BullishDiv OR BearishDiv;
AddColumn( BullishDiv, "Bullish Divergence", 1.0, 
       colorDefault, IIf(BullishDiv, colorGreen, colorDefault ) ); 
AddColumn( BearishDiv , "Bearish Divergence", 1.0, 
       colorDefault, IIf(BearishDiv , colorRed, colorDefault) );

- Tools -> Send to Auto-analysis
- Apply to: All Symbols, N last quotations = 1
- press EXPLORE

Tools -> Send to Auto-analysis- Apply to: All Symbols, N last quotations = 1- press EXPLORE

A different approach can use linear regression instead:
 
// 10 day linear regression slope of close
PriceUp = LinRegSlope( C, 10 ) > 0 ; 
// 10 day linear regression slope of MACD histogram
MacdUP = LinRegSlope( MACD() - Signal(), 10 ); 

Filed by support at 6:54 pm under AFL, Indicators
No Comments