Historical Data Setup
Contents
Data Formats:
Trading Blox supports MetaStock and ASCII text formats. Sample data is ASCII since it is humanly readable and easier to debug.
Note:
MetaStock formatted files do not include the Unadjusted Close price, or the Delivery Month for Futures data. They do not allow the use of extra fields to be added, their values imported and accessible to Trading Blox.
Comprehensive data setup tutorials:
CSI Data: https://www.tradingblox.com/tradingblox/howto-ua.htm
Data File Types:
Metastock:
With the Metastock format you don't need to deal with file names. If Trading Blox can read the Metastock Master file, it will locate all the files in the directory. For Futures, be sure that the symbol is correct in the Futures Dictionary.
Futures and Forex - (ASCII Text):
Futures or Forex Dictionaries require file names assigned to each symbol match the Dictionary symbol in its specified folder.
When the file name starts with the symbol letters, you can often find the file using the Futures Dictionary "Find Symbol" button.
Stocks - See CSI_Symbol_Num for Details:
Stock file specification: ^Top |
CSI ASCII/EXCEL Setting: |
File Name Example: |
|---|---|---|
<symbol-letters>.<suffix> |
|
AAPL.CSV or AAPL.TXT |
<CSI>. <suffix> |
|
5902.TXT |
<symbol-letters CSI#>. <suffix> |
 CSI Stock Name & Stock Number Reference
 New CSI's UA Version Auto-Add _ Character Older US version require the "_" character. New versions of UA add it automatically. Leave the "_" symbol out see if UA automatically adds it for you. |
AAPL_5902.TXT ".csv" is OK It is best to use S# for stocks to avoid the wrong company data issue when a symbol is reused. |
Suffix: Both ".txt" or ".csv" suffix designators are allowed. "CSV" files open into a spreadsheet when double-clicked. |
||
Notes: ^Top Stock Dictionaries can be automatically updated using the main screen's Tool -> Build Stock Dictionary selection. When a Stock Dictionary is built, it will include all the stock symbol files in the selected "Stock" folder. It will also include all the stock symbol files that are contained in all the folders that are in the selected "Stock" folder. To select a stock folder destination that is different from where your stock files are being placed, the
When a Stock Dictionary is being built, it will first look to find that Symbol, or CSI #, in the Unfair Advantage "sdbfacts.csv" database file. During the building of the dictionary any symbols that have duplicates in the database file will not be included in the finished dictionary unless those symbols have a CSI # appended to its symbol, i.e. AAPL_5902.CSV.
Symbol file names can have a ".CSV" or ".TXT" suffix identifier.
When defining the field columns in the CSI Unfair Advantage software, be sure to use an upper case 'V" for the volume column. A Lower case "v" is intended to Futures files that are intended to use the contract's volume. An upper case "V" for volume in a Futures' file selects the total volume value created by adding the volume values from all of the active contracts on a record date. |
||
ASCII Data Files: ^Top
ASCII formatted files must be comma delimited. Dates must be in sequential order where the earliest date information is the first record, and the latest date is the last record in the file.
Daily Data Columns & Price Adjustments:
All fields must be set up as follows. A header record is optional for daily data.
Instrument Type: ^Top |
Daily Data Columns |
||||||||
|---|---|---|---|---|---|---|---|---|---|
Futures: ^Top |
YYYYMMDD, Open, High, Low, Close, Volume, Open Interest, Delivery Month (YYYYMM), Unadjusted Close, Extra Data 1 ... 8
It is recommended that Futures contracts be created using the Back-Adjusted process. Futures Back-Adjustments removes the price gap between expiration months on the date of when the expiring contract is added to the next forward contract.
Roll calculations can use the Delivery Month information when that column is added after the Open Interest column. Price fields of Open, High, Low and Close are the prices that get changed during a price adjustment process.
When prices are back-adjusted, the date where prices are adjusted will be different than they were on that date when the prices were originally active in the exchange. This means adjusted price values on previous dates after an adjustment will not create the same calculation result unless the price adjustment is removed when that date's adjustment price is applied to a calculation. This reason is why we recommend the data file contain the UnAdjusted Close price column. This price column will have the actual price for each date available and it can be used to discover the actual price for each of the four other price fields. |
||||||||
Forex: ^Top |
YYYYMMDD, Open, High, Low, Close, Volume, Unadjusted Close, Extra Data 1 ... 8
Forex files don't use price adjustments. All Forex records are assumed to be accurate transaction values for each record date. In addition, Forex files should be updated daily so the current exchange rate information is available during testing. |
||||||||
Stocks: ^Top |
YYYYMMDD, Open, High, Low, Close, Volume, Unadjusted Close, Extra Data 1 ... 8
Stock Splits: Most stock files over time experience stock splits. Each Stock split changes the price and the number of outstanding shares. Stocks that have Dividends also affect the value of a share. When a stock is split it will show a gap in price that isn't suitable to computer based calculations. This failure happens because the change in number of shares that changes the price, isn't a change that was caused by an positive or adverse economic view of the company's stock.
In order for computer based calculations to be kept accurate, and provide transactions that reflect the economic conditions of how a stock's is viewed, stock split calculation should adjust the prices in the file so the change in shares is reflected in the price. This is important because if the change in price is not adjusted in the data, it is likely to cause testing to create an order transaction where it should not happen. False order events like this can be avoided by back-adjusting the change in price on the date the change shares takes place.
When prices are back-adjusted, the date where prices are adjusted will be different than they were on that date when the prices were originally active in the exchange. This means adjusted price values on previous dates after an adjustment will not create the same calculation results unless the price adjustment is removed when that date's adjustment price is applied to a calculation. Price adjustments can be removed in testing by including the unadjusted close price in the data file. CSI data provides this price when the file creation settings in their software adds UnAdjustedClose price column to the data file (see the stock price fields recommended above).
When a Stocks or Futures file contains the data column UnAdjustedClose, that field value can be used to remove the price adjustment so the calculation result will be the same as it was on the date the exchange created that price record. When it is available in the data, that price in a price adjusted data file can be used to calculate the split ratio and the split ratio direction. By knowing the split ration the basis for adjusting the price becomes available and the transactions that happen during a period where the price had to be adjusted, will be based upon how it would have at that time when there was no price adjustment.
Dividends: When a dividend is added to a stock it increases the price of the stock. Trading Blox can use CSI dividend information to build dividend files for all stocks that have dividends. It builds dividend files that are visible in its Portfolio Editor's Instrument Symbol list section.
Trading Blox Stock File Search Structure Process: Stocks are the only instruments that can have multiple dictionaries when it comes time to locate stock symbols. This means that when Trading Blox starts, it looks for files using the path settings established in the Preferences Data and Folder section. In each folder and each symbol it finds it will add it to a list of symbols that it will use to determine if it can be in a Stock Dictionary, or when a stock dictionary is not used, it will be displayed in the Portfolio Editor's Instrument Symbol list section.
In this next set of images, the path is set to D:\Data\Stocks. That path has five sub-folders and a large number of symbol files. All the files in all the folders, plus all the symbols in the Stock Set's portfolio files will be record so they can be used in a Stock Dictionary, when that is the process being created, or as a list of symbols in the Portfolio Editor's Instrument Symbol list section.  Example Stock Folder and Files Structure Example
 Stock Set Default Stock Portfolio Example All the files and folders in all the places shown in the example above, which includes the symbols listed in the Stock Set Portfolio files, will be listed in the Instrument Symbol list section when a Stock Dictionary is not used. When a dictionary is being assembled, the same search structure will be used to locate symbols.
If you do not have a Stock Dictionary (stockinfo.txt or stockinfo.csv file), it will look for a Metastock MASTER file with that name. If it finds the MASTER file, it will assume Metastock data and use that format process to read the data. If it does not find a MASTER file, it will assume ASCII Text files and look for all the .txt or .csv files in the specified folder, and all the folders contained with the top level Stock folder. Once all the folders have been searched, it will then search all the Stock set portfolio files and collect the symbols in those files.
When there isn't a stock dictionary file, and the above symbol information is done its symbol search, Trading Blox will create a list of symbols from the file names and set files and display that list in the Stock option of the Portfolio Editor symbol listing.
If a symbol is listed in a Stock file portfolio file, and there is no data file with the same name, that item in the list will be unable to load that symbol name information. Trading Blox flags portfolio symbols that don't have data files when it is first started and creates a warning message in the main screen's Log Window display area when the Log Window is enabled.
|
||||||||
NOTE: ^Top All fields after the Date, Open, High, Low, Close (DOHLC) are optional. Files that only contains DOHLC are fine, but some features of Trading Blox will not be effective when the information, like "Delivery Month", or "UnAdjustedClose" are not in the file. |
|||||||||
Adding Data Columns: ^Top
When a need to have more data columns for in a file is required, an additional 8-columns of information can be added to each date record after the file has been updated by your data service.
Trading Blox can provides access to additional data columns. However, access to additional data columns might require additional commas ahead of the extra data so that the information expected by Trading Blox isn't in the wrong place.
Futures information will only access the extra column information after the first 9-columns of specified information is found. Stock and Forex files will only access the extra column information after the first 7-columns of specified information is found. If you are going to add extra columns to a Futures file, but don't include columns for Open Interest and Delivery Month, the file structure will will need to look like this:
YYYYMMDD, Open, High, Low, Close, Volume, Open Interest, , , Extra Data 1 ... 8
When this requirement is followed, Trading Blox will be able to access data in the extra columns because the Futures record found the first nine commas, and could then read the extra column information into the extra column properties for each date record in the file.
CSI Data Updating Service: ^Top
CSI's UnFairAdvantage Updating service will support all the data type requirements shown.
When an UnFairAdvantage instrument portfolio of symbols is created for downloading each day, the following images of UnFairAdvantage's ASCII/Excel Fields will populate the data records with the above Trading Blox data requirements. Less columns can be used when Extra-Data columns are not going to be added after the file updating has completed.
CSI's UnFairAdvantage Portfolio Data Letter Requirements: ^Top
Futures: |
Forex: |
Stocks: |
|---|---|---|
 CSI's Futures data column letters. |
 CSI's Forex data column letters. |
 CSI's Stock-Type data column letters. |
Intraday Data Columns:
For intraday data, a header must be used with at least the first two items "Date,Time", such as:
Date,Time,Open,High,Low,Close,Volume
Thereafter the format of the data is: ^Top
Futures:
YYYYMMDD, HHMM, Open, High, Low, Close, Volume, Open Interest, Delivery Month (YYYYMM), Unadjusted Close, Extra Data 1 ... 8
Stocks and Forex:
YYYYMMDD, HHMM, Open, High, Low, Close, Volume, Unadjusted Close, Extra Data 1 ... 8
Trading Blox accepts the following optional date formats: YYYYMMDD, YYMMDD, MM/DD/YYYY, MM/DD/YY
Trading Blox accepts the following optional time formats: HHMM, HH:MM
Data Dictionary File Names: ^Top
Stocks: StockInfo.txt
Futures: FuturesInfo.txt
Forex: ForexInfo.txt
See Data and Folder Preferences to change data's location folders.
In order for Trading Blox to read your data, there needs to be a dictionary file for Futures and Forex data. This Dictionary file can be in the default Trading Blox Data/Dictionary folder, or it can be in the same folder as the data files. These Dictionary files contains the market specific symbol information for each symbol. The information in the Futures Dictionary is the data that is used to calculate the net outcome value of each Futures Trade.
Trading Blox Futures Symbol Dictionary Tablee
If you don't have a Futures, or a Forex Dictionary available, Trading Blox will display data error information in Log Window, if the Log Window is open, showing no files could be found.
Stock Dictionary:
If you don't have a Stock Dictionary in your stock data folder, Trading Blox will look for a Master file. If found, it will assume Metastock format, and if not found it will assume Text format. If text format, it will load all the files ending in ".txt" or ".CSV' into Trading Blox. When no Stock Dictionary is used the following columns displayed in a Stock Portfolio display will not show
Stock files with a Dictionary:
Stock Dictionary Data Information
Stock files without a Dictionary: ^Top
Stock Data Information without a Dictionary.
When a Dictionary is listed, but one or more of the symbols listed in the selected portfolio are not listed in the Dictionary, an error record will be displayed in the main screen's Log Window area:
Log Window Menu Controls
Missing symbol information will show symbols selected in the portfolio, but not found in the dictionary. Any portfolio selected symbols not included in a Dictionary, when a Dictionay is present, will be shown near the top of the portfolio list without any of the usual Dictionary information. Symbols listed at the top that appear like what is shown next, will not be loaded and available during testing:
The rates files must be located in the Forex Rates folder, which is in the Data folder. Rate files are required for Forex cost of carry, and for the system wide base currency only, earning interest and paying margin. For example, if the base currency is USD, the USD_Rates.txt file will be used for earning interest on excess cash as well as paying margin on stocks on a negative cash balance. The Lending Rate is used when earning interest, and the Borrowing Rate is used for paying margin.
The file can be sparsely populated, and for any given test date, the last available date in the file will be used for the rates. The format for this file is as follows: Date, Lending Rate, Borrowing Rate. If only one rate is in the record, it is the Lending Rate, and the Borrowing Rate is the Lending Rate plus .5%. Note that the concepts of Borrowing and Lending are from the consumer's perspective.
All rate files use a name like "XXX_Rates.txt" or "XXX_Rates.csv". All rate files must be in the Forex Rates folder in the Data folder. The characters "XXX" to the left of "_Rates.txt" part in the example above is the ISO currency code required to find the correct rate file so the use of the information for calculation like computing cost of carry will be accurate. Be sure to include all required ISO currency codes in the ISOCurrencies.csv file location in the Dictionaries folder.
Example Forex Carry Rate File:
Date, Lend, Borrow
20081008,0.015,0.0205
20081029,0.010,0.0155
20081031,0.0045,0.0155
20081216,0.0015,0.0125
Column Descriptions:
Date |
Lending Rate |
Borrowing Rate |
|---|---|---|
20081008 |
0.015 |
0.0205 |
20081029 |
0.010 |
0.0155 |
20081031 |
0.0045 |
0.0155 |
20081216 |
0.0015 |
0.0125 |
Exchange Definition File: ^Top
The exchanges.csv file contains the abbreviation and description of all the exchanges that are used in Trading Blox. A warning will appear if the exchange is used by a market, but missing from this file. To add an exchange to this file simply open the file in excel or notepad, add the exchange and description, and save. Restart Trading Blox.
Or you can press the Examine button next to the Exchange dropdown in the Futures Dictionary. The exchanges are the same for futures and stocks, so this will work for both. The Exchange.csv file is located in the Dictionaries folder, as defined in preferences, default is Trading Blox/Data/Dictionaries.
This Exchange Definition file is a user maintained file.
Currency Definition File: ^Top
The ISOCurrency.csv file is located in the Data/Dictionaries folder and contains all the currencies that can be used by Trading Blox. The file has a version header, column header, and record format is comma delimited.
Version,7
Currency Code,Currency Description
ADF,Andorran Franc
ADP,Andorran Peseta
AED,United Arab Emirates Dirham
AFA,Afghanistan Afghani
ALL,Albanian Lek
Links: |
|---|
|
See Also: |
|
Edit Time: 6/11/2020 10:17:23 AM |
Topic ID#: 155 |