Japan - JGB Yields–More Lattice Charts

This blog is littered with posts about Japan. In one sentence, I think Japan presents opportunity and is a very interesting real-time test of much of my macro thinking. Proper visualization is absolutely essential for me to understand all of the dynamics. The R packages lattice and the new rCharts give me the power to see. I thought some of my recent lattice charts might help or interest some folks.

Get and Transform the Data

# get Japan yield data from the Ministry of
# Finance Japan data goes back to 1974

require(xts)
# require(clickme)
require(latticeExtra)

url <- "http://www.mof.go.jp/english/jgbs/reference/interest_rate/"
filenames <- paste("jgbcme", c("", "_2010", "_2000-2009", 
    "_1990-1999", "_1980-1989", "_1974-1979"), ".csv", 
    sep = "")

# load all data and combine into one jgb
# data.frame
jgb <- read.csv(paste(url, filenames[1], sep = ""), 
    stringsAsFactors = FALSE)
for (i in 2:length(filenames)) {
    jgb <- rbind(jgb, read.csv(paste(url, "/historical/", 
        filenames[i], sep = ""), stringsAsFactors = FALSE))
}

# now clean up the jgb data.frame to make a jgb
# xts
jgb.xts <- as.xts(data.matrix(jgb[, 2:NCOL(jgb)]), 
    order.by = as.Date(jgb[, 1]))
colnames(jgb.xts) <- paste0(gsub("X", "JGB", colnames(jgb.xts)), 
    "Y")

# get Yen from the Fed
# getSymbols('DEXJPUS',src='FRED')

xtsMelt <- function(data) {
    require(reshape2)

    # translate xts to time series to json with date
    # and data for this behavior will be more generic
    # than the original data will not be transformed,
    # so template.rmd will be changed to reflect


    # convert to data frame
    data.df <- data.frame(cbind(format(index(data), 
        "%Y-%m-%d"), coredata(data)))
    colnames(data.df)[1] = "date"
    data.melt <- melt(data.df, id.vars = 1, stringsAsFactors = FALSE)
    colnames(data.melt) <- c("date", "indexname", "value")
    # remove periods from indexnames to prevent
    # javascript confusion these . usually come from
    # spaces in the colnames when melted
    data.melt[, "indexname"] <- apply(matrix(data.melt[, 
        "indexname"]), 2, gsub, pattern = "[.]", replacement = "")
    return(data.melt)
    # return(df2json(na.omit(data.melt)))

}

jgb.melt <- xtsMelt(jgb.xts["2012::", ])
jgb.melt$date <- as.Date(jgb.melt$date)
jgb.melt$value <- as.numeric(jgb.melt$value)
jgb.melt$indexname <- factor(jgb.melt$indexname, levels = colnames(jgb.xts))

Favorite Plot - Time Series Line of JGB Yields by Maturity

p2 <- xyplot(value ~ date | indexname, data = jgb.melt, 
    type = "l", layout = c(length(unique(jgb.melt$indexname)), 
        1), panel = function(x, y, ...) {
        panel.abline(h = c(min(y), max(y)))
        panel.xyplot(x = x, y = y, ...)
        panel.text(x = x[length(x)/2], y = max(y), 
            labels = levels(jgb.melt$indexname)[panel.number()], 
            cex = 0.7, pos = 3)
    }, scales = list(x = list(tck = c(1, 0), alternating = 1), 
        y = list(tck = c(1, 0), lwd = c(0, 1))), strip = FALSE, 
    par.settings = list(axis.line = list(col = 0)), 
    xlab = NULL, ylab = "Yield", main = "JGB Yields by Maturity Since Jan 2012")
p2 + layer(panel.abline(h = pretty(jgb.melt$value), 
    lty = 3))

From TimelyPortfolio

Good Chart but Not a Favorite

As you can tell, I did not spend a lot of time formatting this one.

p1 <- xyplot(value ~ date | indexname, data = jgb.melt, 
    type = "l")
p1

From TimelyPortfolio

Another Favorite - Yield Curve Evolution with Opacity Color Scale

# add alpha to colors
addalpha <- function(alpha = 180, cols) {
    rgbcomp <- col2rgb(cols)
    rgbcomp[4] <- alpha
    return(rgb(rgbcomp[1], rgbcomp[2], rgbcomp[3], 
        rgbcomp[4], maxColorValue = 255))
}

p3 <- xyplot(value ~ indexname, group = date, data = jgb.melt, 
    type = "l", lwd = 2, col = sapply(255/(as.numeric(Sys.Date() - 
        jgb.melt$date) + 1), FUN = addalpha, cols = brewer.pal("Blues", 
        n = 9)[7]), main = "JGB Yield Curve Evolution Since Jan 2012")

update(asTheEconomist(p3), scales = list(x = list(cex = 0.7))) + 
    layer(panel.text(x = length(levels(jgb.melt$indexname)), 
        y = 0.15, label = "source: Japanese Ministry of Finance", 
        col = "gray70", font = 3, cex = 0.8, adj = 1))

From TimelyPortfolio

Replicate Me

code at Gist

Old Post with New d3 Life–GARCH and MA Performance

Parallel coordinates become much more useful when they are interactive, so I recreated one of my favorite blog posts "Trend is Not Your Friend" Applied to 48 Industries and convert the chart to a living breathing d3 parallel coordinates chart courtesy of Ramnath Vaidyanathan's rCharts and Kai Chang's d3.parcoords. I will convert the scatter and horizon plots in a later post.  See here for the finished product.  Here or embedded below is the quick Youtube video showing the creation process.

Changing The Presidential Election with R in the Browser

After I finished with the tutorial post d3 <- R with rCharts and slidify and then saw R creates d3/javascript charts in Ipython Style Notebook, a light clicked.  I could finally answer the lingering question I have had ever since I saw the NYT 512 Path to the White House brilliantly created by Mike Bostock and Shan Carter with d3.

What would happen if different states had a different number of votes?

I never dreamed that I would be able to answer my own question by live R coding in the browser in an Ipython style notebook.  I felt like I needed a screencast to prove it.  Here is the short version.

 

 

Here is the extended version with director commentary (in type).

 

 

 

Thanks Ramnath Vaidyanathan, Yihui Xie, and Mike Bostock.

Production Quality Report with R and knitr on Yen

Sometimes I actually use my experiments for real work.  For example, I wanted to send an update  on the Japanese Yen.  This was a great opportunity to use the chart created in Shading and Points with xtsExtra plot.xts.

I was fairly pleased with the finished product (click here if embed does not show).

R sweave file from GIST:

One Pager Performance Report with knitr, R, and a Different Font

Although I suffer from complete ignorance of typography, with a little help from a post from Hyndsight and post from mages' blog, I wanted to try a different font on the one-pager performance report that we created in Onepager Now with knitR. I do not think Open Sans Light is the best choice for this report, but since it is the most popular font on Google I figured I could not be criticized too heavily by those more knowledgeable than me in typography.

options(tikzDefaultEngine = "xetex")

require(knitr)

knit2pdf("onepager_text_graphics-knitr.rnw", compiler = "xelatex")

And the result (go to Box if the embed does not appear below):

R code from GIST:

Onepager Now with knitR

Since at some point I had trouble with a conflict between knitr and the latex package textpos, I used the lesser Sweave in Another Experiment with R and Sweave.  I ran the Sweave2knitr command and discovered that textpos and knitr play well together now.  Here is the result using knitr (go to https://www.box.com/s/4nftk6qpa0cugapmncsn if the embed does not show below):

.rnw source file from Gist

knitr Performance Report 4

please see knitR Performance Report 3 (really with knitr) and dprint, knitr Performance Report–Attempt 3, knitr Performance Report-Attempt 2 and knitr Performance Report-Attempt 1

Here is another iteration of the ongoing performance reporting attempts using R, knitr, and latex.  In this version, I used the dev=”tikz” option (don’t like but show for an example) and work hard with lattice to get what I believe is a very nice overview page.  I’ll revert back to my second example and use the tufte-handout package for a little cleaner and more modern layout. These examples are intended to stimulate thought and comment. Please let me know what you think.

R code in GIST ( use the line knit2pdf(“pathtofile.rnw”) ) to create the pdf:

knitR Performance Report 3 (really with knitr) and dprint

please see knitr Performance Report–Attempt 3, knitr Performance Report-Attempt 2 and knitr Performance Report-Attempt 1

alstated’s asked a very good question in his comment on knitr Performance Report–Attempt 3, and I’m not sure I could have answered well until I endured some frustrations with Sweave.  I actually did not use knitr for that report, and I struggled with many of the issues that knitr addresses.  knitr’s power comes in its extra ability to control output with additional chunk options described in http://yihui.name/knitr/options, so no more wide, textblock, etc. latex commands.

A very prominent R Finance contributor also alerted me to the dprint (be aware in pre-Alpha) package from Carl Brickner https://r-forge.r-project.org/projects/tabular/ presented at userR! 2010 and also at .  To use, you will have to manually install with the command:

install.packages("dprint", repos="http://R-Forge.R-project.org")

For the final pdf, use the knit2pdf command from knitr:

knit2pdf(“pathtofile.rnw”)

I was delighted with the result and probably will abandon the Sweave-direct option.  Thanks to Carl for dprint and Yihui for knitr.  If you do not see the embedded pdf below, please get directly through https://www.box.com/s/416b473f6c92c581e343.

R code from GIST:

knitr Performance Report–Attempt 3

please see knitr Performance Report-Attempt 2 and knitr Performance Report-Attempt 1

Since the time of my last reporting post, RStudio, knitr, and Sweave have worked extremely hard to make document creation easier by becoming even more streamlined and cooperative (New Version of RStudio (v0.96)).  Thanks so much for these extremely helpful and generous contributors.

Over the years, I have developed an outline for client reports that I think applies in most situations in the money management world.  This also fits for performance reporting and marketing, except the transaction history section would either be eliminated or replaced by a section highlighting trades or ideas that have been used in the past.

From TimelyPortfolio

I thought I would have a completed product before displaying progress, but I just could not wait.  I started working on the Return section.  Please let me know what you think as your comments will guide the rest of the report.  If you are viewing in a RSS reader, you might need to view through this link, since the embed might not work.

R code from GIST:

knitr Performance Report-Attempt 2

Over the years I have changed my learning process from reading thoroughly first before proceeding to reading minimally and then applying immediately.  I very quickly see the gaps in my knowledge.  This method is far more painful but seems to quicken progress up the learning curve.  You will definitely see the process in its entirety with this series on latex, knitr, and performance reporting.  For those who are still hesitating, jump right in with me and let me know your result.

As I experimented with Attempt 2, I could see from others sample .Rnw files that LyX can serve as the IDE for Sweave/knitr documents, and I thought this would be a more pleasant route.  However it was not, and I eventually reverted back to old-school manual coding.  This post from R, Ruby, and Finance talks about TeXnic Center as an IDE.  Maybe I will try it for attempt 3. I am still a long way from a useable result (any bets on how many attempts it takes?), but the report is definitely an improvement

In Attempt 2, I used the echo=FALSE option to not output the R code on the final pdf report.  However, the code is still all there for public view within the .Rnw file.

I just remembered that ttrTests offers a Latex output option. Just what I need--another distraction. Look for that in future posts.

R code from GIST:

knitr Performance Report-Attempt 1

I get very excited about new R packages, but rarely is my excitement so fulfilled as with knitr.  Even with no skill, I have already been able to adapt the example Yihui Xie provides in his knitr Graphics Manual into a crude first version of a performance report that I could actually show clients and prospects.  Although this is far from production quality, two days of experimentation already has me to a level that assures me of the wonderful potential of combining the existing amazing R packages with knitr.  Before knitr, I do not believe I could have accomplished even this rough first draft.

If you have not played with MiKTeX before, you will need to use the Package Manager to install the xcolor and tufte-handout templates for this example to work properly.  MiKTeX is installed automatically with LyX as discussed in yesterday’s post Latex Allergy Cured by knitr.  If xcolor causes problems, then just change your repository to another repository.

knitr weaved with a little PerformanceAnalytics, lattice, and latticeExtra provides this first draft.  As always, thanks so much to the brilliant contributors that have made this possible.

R code in GIST:

Latex Allergy Cured by knitr

I have always known that at some point I would have to succumb to the power of Latex, but Latex has been uncharacteristically intimidating to me.  I finally found the remedy to my Latex allergy with the amazing and fantastic knitr package from Yihui Xie.  With very minimal effort, I ran my first experiment and now am extremely excited to incorporate it in production-quality performance reports (I plan to document the steps to get there in future posts).

For those starting from scratch on Windows, I think the easiest method to get up and running is to install LyX, which will also install MikTex.  If these are successfully installed, then you should be ready to experiment with knitr in R.

I will use knitr’s stich function, which is clearly not designed for the robust production use of knitr, but makes for a very easy first test.  stitch will open a very short script, apply a template, and generate a Sweave style .Rnw (can be changed).  knit2pdf converts the .Rnw file into a pdf, and with a couple lines of code you get a remarkable result.

 

 

R code from GIST (unbelievably only 7 lines):