Problem getting our data into OpenGamma

OG-Platform
1

I have the installation and the site working. Now I am trying to import our data files into the system. I was able to generate the template files for various security types. So when I tried loading equity positions data using the generated template and our data, I get this message:

“Portfolio upload requires Bloomberg data
Please see Examples-Bloomberg”

To isolate the issue I just copied the following rows from your documentation section “Portfolio and Position data connectivity->Command Line tools -> Loader tool-Examples” and still get the same error, please advise.

Also if I try to create a new security type, I get this error:

“Status: 500 Internal Server Error; Message: This is a placeholder security loader. For fully supported implementations supporting major data vendors like Bloomberg and Thomson-Reuters, Please contact sales@opengamma.com.
”

Thanks,
Aditya

2

So, it sounds like you’re using the ‘Examples’ package. This contains some pre-packaged security reference data, simulated market data and generated time series. We aren’t in a position to ship an exhaustive reference data set for licensing reasons so you really need some internal or external source of reference, live and historical data to get up and running. The upload format you’re using (the simple 5-6 column format, referenced as option 4 here), requires access to a Bloomberg terminal or SAPI installation because it only provides the tickers and uses our loaders to pull security reference and historical data in from Bloomberg, apologies that it isn’t clearer.

To run without Bloomberg you need to use one of the more full data import formats, either the zip or the xml format detailed on the web site (click the appropriate child page). You’ll then need to import historical data (from which you can run views using e.g. close data) or create market data snapshots. The tools for importing/exporting snapshot data should be in the upcoming 2.1 release coming in early September. There is a command line historical time series import tool you can use in the meantime (see http://docs.opengamma.com/display/DOC200/Using+the+Historical+Time+Series+Loader+Tool). We’re also working on integration with the website http://www.quandl.com which provides free data sets. I’m not sure if this will make the 2.1 release though and will likely have to wait until 2.2 in November(ish).

I hope that’s of some help.

3

For full data import formats like the zip format what do you recommend to use for importing the data - through the web interface or through the command line tools?

What kind of account setup is needed with bloomberg? We do have an account with Bloomberg which we use to download market data automatically on a daily basis. How do we use that account to plug in the Open Gamma platform?

4

Regarding the data import, it shouldn’t matter if you use the web import or the command line, but the command line might give you more control I suppose (and allow you capture any logging output in case there are errors).

If you install the demo on a machine with a Bloomberg terminal, it should automatically be detected and you’ll get a different set of example views and configurations tailored towards working with Bloomberg identifiers etc, rather than the non-Bloomberg demo, which for historical legal reasons uses an identifier scheme (by that I mean e.g. Bloomberg ticker + market sector code such as AAPL US Equity) that we made up internally called OG_SYNTHETIC_TICKER.

5

I downloaded examples.zip from your site and tried importing the zip file format through the web interface and still get the same bloomberg error.

Regarding the bloomberg option we have an ftp system account with them which I want to see if I could use. Is using the bloomberg terminal only option for connectivity with bloomberg, since generally these terminals are installed on machines used by traders, risk analysts, etc ?

6

We don’t currently have support for Bloomberg Data License, but we’d like to. The reason we don’t is that we don’t currently have access to it as part of our developer license but we continue to try to get access. I’m sorry you’ve had problems with the examples.zip file. I’ve opened a ticket to investigate PLAT-4484

7

If we don’t have bloomberg terminal but want to leverage the market data we store ourselves, what interfaces do we need to implement to get it working?

Also if we need to get in our security and position data, I see some documentation on using interface SecuritySource and section “Writing a Portfolio or Position Feed”, but how do we integrate these with the main engine and configure the engine to run with them.

8

So if I get it right, I can’t use the csv format to create a portfolio of stocks if I don’t have a Bloomberg connection?
I am asking that because I just saw that the xml format is not supporting simple stocks portfolio.
(I am always talking about the examples-simulated package).

By the way I would really like to see a full example of loading a portfolio that I have created using csv files in your examples-simulated package, because right now all the values of the portfolio, positions and historical data are hardcoded using java code and random functions.

Thank you.

Stavros Kefaleas
Software Developer at The Markets Trust company

9

In the context of my previous question I would like to know what exactly each of these scripts located at /examples/examples-simulated/target/server-dir/scripts does.
Thank you.

Stavros Kefaleas
Software Developer at The Markets Trust

10

I am trying to use the portfolio-loader-tool.sh from the examples-simulated package.
The csv file I am using has been taken from your documentation pages(http://docs.opengamma.com/display/DOC200/CSV+and+XLS+file+formats) and it is the following:

Equity.csv :

"companyName","currency","exchange","exchangeCode","externalIdBundle","name","position:quantity","securityType","shortName","trade:counterpartyExternalId","trade:deal","trade:premium","trade:premiumCurrency","trade:premiumDate","trade:premiumTime","trade:quantity","trade:tradeDate","trade:tradeTime"
"OCCIDENTAL PETROLEUM CORP","USD","NEW YORK STOCK EXCHANGE INC.","XNYS","CUSIP~674599105, ISIN~US6745991058, OG_SYNTHETIC_TICKER~OXY","OCCIDENTAL PETROLEUM CORP","320","EQUITY","","CPARTY~BACS","","","","","","320","2010-12-03",""
"EXXON MOBIL CORP","USD","NEW YORK STOCK EXCHANGE INC.","XNYS","BLOOMBERG_BUID~EQ0010054600001000, BLOOMBERG_TICKER~XOM US Equity, CUSIP~30231G102, ISIN~US30231G1022, SEDOL1~2326618","EXXON MOBIL CORP","4330","EQUITY","XOM","CPARTY~BACS","","","","","","2000","2010-12-01",""
"CONOCOPHILLIPS","USD","NEW YORK STOCK EXCHANGE INC.","XNYS","CUSIP~20825C104, ISIN~US20825C1045, OG_SYNTHETIC_TICKER~COP","CONOCOPHILLIPS","2740","EQUITY","","CPARTY~BACS","","","","","","2000","2010-12-01",""
"MURPHY OIL CORP","USD","NEW YORK STOCK EXCHANGE INC.","XNYS","CUSIP~626717102, ISIN~US6267171022, OG_SYNTHETIC_TICKER~MUR","MURPHY OIL CORP","430","EQUITY","","CPARTY~BACS","","","","","","430","2010-12-03",""
"CHEVRON CORP","USD","NEW YORK STOCK EXCHANGE INC.","XNYS","BLOOMBERG_BUID~EQ0010031500001000, BLOOMBERG_TICKER~CVX US Equity, CUSIP~166764100, ISIN~US1667641005, SEDOL1~2838555","CHEVRON CORP","260","EQUITY","CVX","CPARTY~BACS","","","","","","260","2010-12-03",""
"HESS CORP","USD","NEW YORK STOCK EXCHANGE INC.","XNYS","CUSIP~42809H107, ISIN~US42809H1077, OG_SYNTHETIC_TICKER~HES","HESS CORP","100","EQUITY","","CPARTY~BACS","","","","","","100","2010-12-03",""
"MARATHON OIL CORP","USD","NEW YORK STOCK EXCHANGE INC.","XNYS","CUSIP~565849106, ISIN~US5658491064, OG_SYNTHETIC_TICKER~MRO","MARATHON OIL CORP","3000","EQUITY","","CPARTY~BACS","","","","","","2000","2010-12-01",""

First I run the server using : ./examples-simulated.sh debug

The after the server is running I am running the following command to persist this portfolio into my postgres database :

./portfolio-loader-tool.sh -c …/config/toolcontext/toolcontext-examplessimulated.properties -f Equity.csv -s Equity -n “Stavros” -w

The tool is reporting that it has persist these data into the database but when I press Load again in the Gui Window of the server I am still getting the hard coded ‘Equity Portfolio’ with 25 Consumer discretionary, 15 Materials an so on.

Any ideas?

11

Are you looking at the admin pages? http://localhost:8080/jax/
It is easiest to see if the data is loaded by looking there, either in the portfolios, positions or securities areas.

12

As a general comment, I found it is better to use the actual server for the “-c” option with any of the data loaders.
e.g… … -c http://localhost:8080
This way you make sure the script is picking up the correct configuration.

13

Hello,

  1. Did anyone manage to use that portfolio-loader tool ? Personnally I have changed the config file %OG-Platform%\examples\examples-simulated\target\server-dir\config\toolcontext\toolcontext-examplessimulated.properties to use the Postgress database (which I am using in the GUI) but the log mentions it can’t find the postgress driver.

  2. I have also noticed that the sample portfolio contains securities identified by ISIN or Bloomberg BUID. My purpose is to load any ‘dummy’ equity e.g HDASW1 which is not quoted on NYSE, not referenced in Bloomberg.
    Is this possible to achieve that without coding and preventing the message
    "This is a placeholder security loader. For fully supported implementations supporting major data vendors like Bloomberg and Thomson-Reuters, Please contact sales@opengamma.com"
    So far, the only “workaround” I see would be bypassing the OG logic and performing direct inserts into the database.

14

Hi all,

Just to let everyone know, I’m doing some work to change the example simulated servers so that you can get a running installation without Bloomberg. The first thing that I’m doing is to refactor the java code a bit so that you can start out with a “blank” server.

As far as getting the postgresql driver working. The easiest way of doing that is to go into the examples-simulated/pom.xml file and then uncomment the dependency that pulls in the postgresql driver. Once you do that, a maven build will pull in the jar files for postgresql.

15

FYI, I just put in pull request #105, which splits up the database populator in the simulated data server into an init server which creates a blank database and then the code which loads in the simulated data.

16

Pull request applied.