Historical data (and how to get it)

Historical price data is used in [Test] and [Train] mode, and can also be used in [Trade] mode for pre-loading the LookBack period. The needed data period is determined with StartDate and EndDate. If historical prices from the desired period are missing, they can be downloaded with the assetHistory function or the Download script (see below). All downloaded prices are usually aligned to UTC time. The broker's time zone does not matter. Therefore it's normally no problem to use price data from different sources for the simulation and for trading.

Zorro stores historical price data in files the History folder. The files can contain additional data such as spread and volume ("level-1 data"), expiration dates, option chains, or order book content ("level-2 data"). The file name begins with the asset name. The file extension indicates the content:

Any file is a list of structs. Any struct represents a candle, quote, contract, or order book entry. The struct size - the number of elements besides the time stamp - is indicated by the number after the "t" extension. Large files can be split in multiple years; in that case the file name has the year number attached. For instance, "AAPL_2010.t2" contains all AAPL order book data from 2010. If additional data is required, it can be imported in arbitrary user-specific formats in a dataset. The .t6 price data files included with Zorro are normally split in years and 1-minute candle based, but any other time base can also be used.

The same asset can have different price histories, for instance EOD data for strategies with daily bars, M1 data for day trading and T1 data for scalping. The historical data files can be distinguished with a postfix character to the file name, f.i. "AAPLeod.t6" for EOD data with no year number, "AAPL_2018m1.t6" for M1 data split into years and "AAPL_2018.t1" for the tick data. The script can then select the right price history through the History variable, f.i. "History = *eod.t6" for selecting the EOD data, "History = *m1.t6" for the M1 data and "History = *.t1" for the tick data.

A .t1 or .t6 price data file is a list of T1 or T6 structs in descending order (newest price first).

typedef struct T1
{
DATE time; // timestamp of the tick in UTC, OLE date/time format
float fVal; // price data, positive for ask and negative for bid
} T1; typedef struct T6 { DATE time; // UTC timestamp of the close, OLE date/time format float fHigh,fLow; float fOpen,fClose; float fVal,fVol; // additional data, like ask-bid spread, volume etc. } T6;

Both .t1 and .t6 data can contain the current ask-bid spread, which is returned by the marketVal function. For backtesting with variable spread, set Spread = marketVal() in the script.

Historical data of options and futures are stored in a .t8 file that includes option specific parameters such as type, strike, and expiration date. It is a list of CONTRACT structs in descending order:

typedef struct CONTRACT
{
  DATE  time;   // time in UTC, OLE date/time format
  float fAsk,fBid; // premium without multiplier
  float fVal,fVol;  // additional data, like open interest, volume etc.
  float fUnl;   // underlying price
  float fStrike; // strike price
  long  Expiry; // YYYYMMDD format
  long  Type;   // PUT, CALL, FUTURE, EUROPEAN, BINARY
} CONTRACT;

Dependent on the History variable, historical price data can be taken either from a .t1, .t6, or .t8 file. The default is .t6. If a .t8 file contains different underlying prices for the same time stamp, which can happen with futures of different expiration, the average of the underlying prices is used for the historical price. Historical data should have the same format for all years and assets; mixing .t6 from 2016 and .t1 from 2015 would produce an error message. The exception are options that are always loaded from a .t8 file, while the underlying prices can come from the same .t8 or from a .t6 or .t1 file.

Historical order book data is stored in a .t2 file that contains the ask/bid quotes and the quoted volume:

typedef struct T2
{
  DATE time;  // timestamp in UTC, OLE date/time format
  float fVal; // price quote, positive for ask and negative for bid
  float fVol; // volume or size
} T2; 

For downloading and reading data files with a resolution of less than one minute (such as .t1 files), Zorro S is required. For converting price data from an external source to the Zorro format, separate it into years, convert it to a list of T6 or T1 structs, and store it in a binary file with the format and name described above. The structs are stored in reverse order, i.e. the most recent tick comes first. The OLE DATE format is a double float value, counting days since midnight 30 December 1899; hours, minutes, and seconds are represented as fractions of days. The timestamps are supposed to be the UTC time of the last price of the tick, i.e. the time of the Close price. If you have price data based on local time, convert it to UTC (you can find some example code here). Daily (D1) price history with timestamps containing the date only (no time of day) is automatically converted. If the original price data contains bid prices, you can convert them to ask prices by adding the spread. If the price data contains time stamps not from the close of a tick, but from the open or middle price, correct them with the TickFix variable if needed for the strategy. Normally it is not relevant for backtest results if ticks contain ask, bid, or trade prices, or if time stamps are from the tick open or the close.  

The fOpen, fClose, fHigh and fLow data streams can be separately accessed in the script. The fVal and fVol data can be used to store and access additional data, such as spread, quote frequency, trade volume, or rollover. Data in .t1 format can be used for additional information, such as external indicators or statistics such as the VIX (in that case set Detrend at 16 for disabling the automatic fixing of price data). All data streams are automatically synchronized to the streams of the first asset call in the script; so it does not matter if some data is only available in daily or weekly 'ticks'. The asset function can be used to select between different data files, while the price functions select between the four data streams inside the file.

Historical data in arbitrary CSV formats can be converted to .t1, .t6, or .t8 with the dataParse function. Exporting or converting historical data to and from other formats is also easy due to the simple format - in the Strategy folder you can find several small scripts for this purpose. Download.c (see below) downloads historical data in .t6 or .t1 format from Yahoo, IB, FXCM, or Oanda. CSVtoHistory converts data from arbitrary .csv formats, f.i. from HistData, Quandl, or Yahoo, to the Zorro .t6 format. Example code for automatically downloading .csv price data from the Internet or from a provider API can be found on the http page. A script for generating .t8 historical data for options from the yield rate, dividends, and underlying price history can be found here. Example scripts for converting history files from complex .csv or .nxc formats to the .t1 format can be found here.

For looking into .t8, .t6, .t2, or .t1 files and displaying their content in a panel, use the History script.

Downloading Historical Data

A large archive of history files is available on the Zorro Download page. For more price histories you can use Download.c, a small script for downloading or updating prices and asset parameters from online sources or from your broker or signal service. You can download either the history of a single asset, or of all assets of a given asset list. The script opens a control panel with toggle buttons (blue) and entry fields (yellow):

History  
AAPL Either asset symbol, or asset list file name
2016-2019 Time period to download
M1 Data resolution: M1, Tick, or EOD
Ask/Bid Data type: Ask/Bid, Trades, Unadjusted, Assetlist
Broker Data source: Broker, Stooq, AlphaVantage
Download! Click here to start the download.

Asset or Asset List

Enter here the symbol of the asset in the form needed by the broker or price source, f.i. "AAPL.US" for downloading AAPL data from Stooq. If the name contains the word "Assets" or ends with ".csv" (f.i. "AssetsFix.csv"), all assets of the asset list with the given name are downloaded.

Data resolution

Data type

Data source

Not all combinations make sense. For instance, you can normally not download M1 data or asset parameters from online data sources. Check with your broker which assets are available. It is recommended to download forex data from the same broker that you use for trading, since it's broker dependent. If a price history file of the same asset already exists, the new prices are appended at the end. If the file was already updated on the same or previous day, no data is downloaded. If you tried to download nonexisting data from a data source, you'll normally get a file containing an error message as response. Zorro won't interpret that file, but you can find it in your History folder as the most recent file, open it with a text editor, and see what's wrong.

 After all is downloaded, click [Stop] to end the session.

See also:

Bars, file, Dataset, asset parameters, assetHistory

► latest version online