Issue 6/2001 (No. 14)
Issue 6/2001. No 14.
4 March 2001.
Copyright (C)2001 Tomasz Janeczko.
All back issues available from:
http://www.amibroker.com/newsletter/
|
Issue 6/2001 (No. 14) |
AmiBroker Tips weekly newsletter. Issue 6/2001. No 14. 4 March 2001. Copyright (C)2001 Tomasz Janeczko. All back issues available from: http://www.amibroker.com/newsletter/ |
Welcome to the 6th issue of AmiBroker Tips newsletter in the 2001. In this
issue I will give you another example on automating the process of downloading
daily updates. This time I will present a script-based alternative to using
AmiQuote.
Just a reminder: if you have any comments/suggestions or article ideas, please don't hesitate to drop a line to newsletter@amibroker.com
In my previous article on automating downloads (see 2/2001 issue of the newsletter) I presented a method that used FileSystemObject to access local files and my URLGet program to download the files from remote hosts. The scripts presented then worked for Australian and Warsaw Stock Exchanges. In this article I will present similar script that can be used for retrieving daily quotes for US and Canada markets.
As you know, AmiBroker comes with AmiQuote downloader program that can be used to download both historical and daily quotes from Yahoo finance site. AmiQuote is very handy when it comes to downloading long histories (5-10 years), but when it comes to daily updates it is not efficient because it requires multiple steps to make the thing done. Some of you were complaining that this task is too complicated and should be simplified. So in this article I will present an alternative to using AmiQuote for daily updates only. For historical data AmiQuote remains recommended tool for obtaining US & Canada quotes.
The daily downloader script for Yahoo is a simple rewrite of the ASX script presented in 2/2001 issue of the newsletter. I assume that you have read that article and you are familiar with URLGet downloader program and you know some basic concepts of Windows Scripting and AmiBroker object model. If not, please read previous issues of the newsletter before continuing with this one.
The script uses the same helper functions and variables as an ASX script. The only thing really different is the way the downloads are handled. In case of ASX script the all daily quotes are available in single file. This makes download very easy - just download and import one file. In case of Yahoo quote server it is not that simple. Yahoo's server takes a ticker or a list of tickers in the the following URL:
http://quote.yahoo.com/d/quotes.csv?s=<list_of_tickers_here>&f=snl1d1t1c1ohgv&e=.csv
and then returns a comma separated file with daily quotes of the tickers given in the URL. AmiQuote uses exactly the same scheme to request data for single ticker at a time, but this is not very efficient because you need to download as many files as the number of tickers. A better way to do is to built the list of tickers and retrieve multiple quotes in single download. Unfortunatelly, you can not ask Yahoo server for 4000 tickers at once, because of the limitations of HTTP GET command that does not allow URLs to be longer than 4096 characters.
So, Yahoo daily downloader script must retrieve the list of tickers from AmiBroker's database, split the list into smaller chunks (for example 30 tickers in one), perform multiple downloads for the chunks of data and multiple imports of downloaded files. And this is exactly what this new script does.
As you can guess the whole script is a little bit complicated so I won't include complete listing here but will focus on just the user definable stuff, so let's take a look at constants that could be changed by the user:
/* The ticker to check */
/* Set it to empty string "" if you don't want to check */
ChkTicker = "^NDX";
/* The folder where the files will be downloaded */
DestDir = "C:\\Yahoo\\";
/* The name and the path to downloader program */
DownloaderPrg = "URLGet.exe";
/* Force download - if true causes downloading data file */
/* even if it exists on the local drive */
ForceDownloadFlag = true;
/* URL from where data are downloaded */
/* quote.yahoo.com supports US and Canada markets only */
URLPrefix = "http://quote.yahoo.com/d/quotes.csv?s=";
URLPostfix = "&f=snl1d1t1c1ohgv&e=.csv";
/* extension of file name, YYYYMMDD will be prepended */
FileExt = ".aqd";
/* Max. number of tickers to download in one run */
/* Should be no more than 30 because of the limitations of command line string length */
ChunkSize = 30;
The script uses C:\Yahoo folder by default for downloaded data. If you don't want to modify the script you should create such folder and copy the URLGet program there. Other data and program locations could be defined by changing DestDir and DownloaderPrg constants. ForceDownloadFlag could be changed to false if you do not want to re-download the files that were downloaded wrong.
Now I suggest you to take a look at complete scripts available here: Yahoo downloader script. You will need also URLGet program available here.
Note about the performance of this scipt: I found that downloading daily updates using this scipt is considerably faster than using AmiQuote for daily downloads, because the script downloads 30 tickers in single run, making the number of downloads considerably smaller. Using a 33kbps modem I can update my ~5000 tickers Nasdaq database in less than 8 minutes.
.... and that's all for this week - hope you enjoyed reading
AmiBroker Tips weekly newsletter. Issue 6/2001. Copyright (C)2001 Tomasz Janeczko. All back issues available from: http://www.amibroker.com/newsletter/