Sample Stat

Hide your password in R

Wed, 20 Dec 2017 10:51:12 +0100

There is several way to hide password in R. I recommend you to see this video if you want to have a more accurate picture on how to do it using fully encrypted password. There is specific libraries available in R which deal with encryption like sodium and also using XOR operator and others.

In my work I had to deal with a fix password like for example "password1234" that the administrator gave me and to enter it in my code when I need to login to the SQL database. In my opinion there is huge security issue using password like this. You can store the code name encrypted on your computer, however, when login, you will need to decrypt it and use the plain text format of your password to have access to the database or anything else. Anyway, I found how to encrypt my plain text password using the library PKI which used to encrypt and decrypt word.

For this tutorial you will need to generate a private key and a public key. The private key will generate a random vector which will be use to decrypt the password. You will need to carefully save it on your computer, otherwise you will cannot decrypt your password. The public key will be use to encrypt the password. Thus, you will use the

library(PKI)
#to be erase after first execution
  # generate 2048-bit RSA key
    key <- PKI.genRSAkey(bits = 2048L)
##
##      # extract private and public parts as PEM
      priv.key <- PKI.save.key(key)
      pub.key <- PKI.save.key(key, private=FALSE)
##      # load back the public key separately
      pub.k <- PKI.load.key(pub.key)
##
##      # encrypt with the public key
      find_me <- PKI.encrypt(charToRaw("password1234"), pub.k) #to remove after saving it in a separate file

find_me

Le chargement a nécessité le package : base64enc
  [1] 73 67 c7 c3 45 9b d4 f2 60 41 23 a0 f6 05 77 43 d2 6f 1c 6f 5f 3d 9b 0e 17
 [26] dd 0a 0d fe 0c 3e 51 6e 64 8b e0 c6 fa da 47 cd af 6e 7f 38 c5 22 d5 bb 6a
 [51] a5 c8 a9 80 5f b9 5e b4 09 1d 52 e2 b5 41 f6 07 b1 bb 01 29 59 91 7a c7 80
 [76] 00 62 4b 4a 48 5a d3 41 48 80 71 3e a8 ab 93 03 6f 20 3a 0a 5f bc bc 56 b5
[101] eb 14 cb b1 ad 41 10 83 81 53 61 41 0f ee a5 bf 51 f9 a9 f3 a5 f5 79 6a dc
[126] 2d b3 05 29 55 79 03 e7 bb f0 b1 28 98 1c 3e 5e f5 c7 d8 f7 42 b8 bb f3 97
[151] 0b 17 96 98 66 f1 ac 20 12 e5 c3 df 47 13 eb 08 85 17 bc e8 ee db 1c b8 47
[176] fe 88 55 df 42 79 4c 61 aa 65 14 46 0d fb dd 5b 73 ad ce 64 43 9b 49 86 8a
[201] 82 bc c2 02 99 e1 72 09 c5 48 3e 3d d8 3c 02 db b8 52 04 7f 7d c4 aa b6 44
[226] c3 c9 93 51 05 0d 93 d4 75 df ec 36 70 9e 40 61 14 d7 13 e6 81 32 55 7d 7b
[251] 0b 64 f3 9d 63 32

This is the result of the encryption of password1234 using the public key.

Then we will need to store this encryption in a file on the computer AND also the private key, otherwise you cannot decrypt the password.

##      # decrypt with private key
##
 save(  priv.key, file = "priv_key") #save private key in a separate file
## # save(  pub.key, file = "pub_key") # no need to save public key
 save(find_me, file = "find_me") #save encryption
##
## #remove object from the R history
 rm( priv.key)
 rm( pub.key)
 rm(find_me)
## #erase above code after first execution

And here starting a fresh session of R to decrypt your password:

#when starting a new session use:
library(PKI)

load("priv_key") #load private key. Cannot be load in the PKI.decrypt function
load("find_me") #load the password to decrypt


rawToChar(PKI.decrypt(find_me, PKI.load.key(priv.key)))#print password - erase this line after making sure this is working

password1234

Here you had your decrypt password.

Mean of the n last or first

Thu, 01 Jun 2017 02:00:00 +0200

A small post today that could be very useful. A colleague ask me how to compute the mean of the 5 oldest trees in a populations. The following code that I have found in stackoverflow.

library(data.table)

treeage <- data.table(age = rnorm(50,200,50), group = rep(LETTERS[1:5],each = 10)) #dataset with trees age in 5 different group


subset_mean_three <- treeage[order(rank(age), group)][,tail(.SD,3),by=group] #subset the 3 oldest trees for each unique group
group_age <- subset_mean_three [, .(age_mean= mean(age )), by = group] #Compute the mean of the tree age from the 3 oldest trees per group

merge(treeage,group_age)[1:15] #adding the mean age of the 3 oldest trees in the treeage data table

As well you can ask for the n quantile age distribution or the youngest trees ages with the following code:

#For the quantile :
subset_quantile  <- treeage[, .(quantile_age = quantile(age,0.90)),by=group] #subset the 90 qauntile age for each unique group

#For the 3 youngest trees:
subset_mean_young <- treeage[order(rank(age), group)][,head(.SD,3),by=group] #subset the 3 youngest trees for each unique group
group_age_young  <- subset_mean_young [, .(age_mean_young = mean(age )), by = group] #compute the mean of the 3 youngest trees for each groups


#merge results
Reduce(merge,list(treeage,group_age,subset_quantile,group_age_young))[1:15] #Merging all the tables together

    group      age age_mean quantile_age age_mean_young
 1:     A 173.3567 303.9826     291.0337       170.4187
 2:     A 262.4292 303.9826     291.0337       170.4187
 3:     A 233.8751 303.9826     291.0337       170.4187
 4:     A 211.6698 303.9826     291.0337       170.4187
 5:     A 193.1739 303.9826     291.0337       170.4187
 6:     A 282.6023 303.9826     291.0337       170.4187
 7:     A 144.7256 303.9826     291.0337       170.4187
 8:     A 366.9163 303.9826     291.0337       170.4187
 9:     A 201.2530 303.9826     291.0337       170.4187
10:     A 206.6334 303.9826     291.0337       170.4187
11:     B 265.5331 292.0307     279.6926       185.2296
12:     B 228.0602 292.0307     279.6926       185.2296
13:     B 200.2322 292.0307     279.6926       185.2296
14:     B 219.2227 292.0307     279.6926       185.2296
15:     B 203.7675 292.0307     279.6926       185.2296

You can also see how to merge multiple data table together using the function Reduce(merge, list(DT1, DT2, DT3, ...)) finding also in stackoverflow.

parallelisation computing

Wed, 10 May 2017 02:00:00 +0200

I already show you how to do for loop and nested for loop. When you want to do parallelisation computing, you simply replace the for command by the foreach command and %dopart%.

Depending if you are on Windows, MacOS or Linux, you will need to do the parallelisation differently. I personally do it on Windows and Linux.

Within a foreach loop, you will need to declare the loop variable as you usually do in the for command, but also to declare the package that you are going to use within the foreach loop using the command .packages. You also need to set the .multicombine to TRUE is you want combine all the parallelism computing within a single output and finally you can use the .final argument to customise the output (list of list, etc.).

Keep in mind that the parallelisation will be processing using the number of core that you declare in the foreach session and those results will be sort in the same First we create 100 random point patterns with 2 different marks "A" and "B" :

#creating random spatial data with 2 marks
library(spatstat)

plot_NA_list <- NULL
for (i in 1:100) {
    # some arbitrary coordinates in [0,1]
    x <- runif(70)
    y <- runif(70)

    # marks
    m <- as.factor(sample(c("A","B"), 70, replace=TRUE))
    #creat ppp with marks    
    X <- ppp(x, y, c(0,1), c(0,1), marks=m)
    X <- list(X)
    names(X) <- i #set the names for each ppp
    plot_NA_list <- append(plot_NA_list,X)#add the ppp to the list
}

plot_NA_list[1:2]

$`1`
Marked planar point pattern: 70 points
Multitype, with levels = A, B 
window: rectangle = [0, 1] x [0, 1] units

$`2`
Marked planar point pattern: 70 points
Multitype, with levels = A, B 
window: rectangle = [0, 1] x [0, 1] units

Before going more further, you will need to install the following package in R: doMC (or doSNOW if you use Windows), spptest and marksummary package which are 2 greats packages when you do spatial point pattern (you can found them at https://github.com/myllym/GET for the spptest package and at https://github.com/myllym/marksummary for the mark summary package).

The following code will compute the "Lest" function overall 30 spatial pattern simulation (usually this is 2499 simulation) from the random data set. The "Lest" function will basically see if the mark "A" from the spatial pattern is randomly distributed regarding the mark "B" across the spatial pattern (this is define in the simulate argument in the envelope function. Then, we compute a rank envelope test (see here for more details). Thus, if the result from the rank envelope test is significant and when we plot it the observed data are above the envelope, then "A" and "B" have a positive correlation relationship (i.e. there are attracted one to another), if this is below the envelope threshold, then "A" and "B" have a negative correlation relationship (i.e. there are segregate one to another).

Finally, we put the result of the rank envelope test into a list with also the random labelling residual data.

library(doMC) #only for Linux,used library(doSNOW) for Windows and uncomment the above lines
#      clusterN <-  detectCores()-1  ### choose number of nodes to add to cluster in doSNOW ; Compute the number of core available from the computer less 1
#      cl = makeCluster(clusterN, rscript="Rscript.exe", type='SOCK')  ###Cluster for doSNOW
core <- detectCores()-1#Compute the number of core available from the computer less 1
registerDoMC(core)#registerDoSNOW(cl) ###for Windows

master_list <- NULL
z <- NULL

NA_list_result <- foreach(z = names(plot_NA_list),.packages = c("spatstat","spptest","marksummary"),.multicombine = TRUE, .final = function(z) setNames(z, names(plot_NA_list))) %dopar% {
    plot_NA_list <- plot_NA_list[[z]]

    ppp_envelope_NA <- envelope(plot_NA_list, fun = "Lest",nsim = 30,correction="translate",savefuns = TRUE,simulate = expression(runifpoint( plot_NA_list$n, win= plot_NA_list$window))) #need to change the simulation to 2499 and to add at the end 'simulate=expression(runifpoint(plot_one_specie$n, win=plot_one_specie$window))'
    ppp_curve_set_NA <- envelope_to_curve_set(ppp_envelope_NA) #change the envelope class in a curve_set class. Need to add the use_theo argument
    ppp_curve_set_NA$theo <-  ppp_curve_set_NA$r
    res_NA <- residual(ppp_curve_set_NA,use_theo = TRUE)
    rank_NA <- rank_envelope(res_NA,lexo = FALSE,savedevs = TRUE) #use to plot and to have result

    master_list <- list(rank_env_NA = rank_NA,random_labelling = res_NA) #,residual_NA=res_NA
    return(master_list)
}  #stopCluster(cl)
###End of parallelisation for Windows and Linux

It will took some time to compute every 100 spatial patterns.

Then you should have this list of result:

NA_list_result[1:2]

$`1`
$`1`$rank_env_NA
Rank envelope test
 p-value of the test: 0.3225806 (ties method: midrank)
 p-interval         : (0.03225806, 0.6129032)

$`1`$random_labelling
curve_set object containing :
List of 4
 $ r          : num [1:513] 0 0.000488 0.000977 0.001465 0.001953 ...
 $ obs        : num [1:513] 0 -0.000488 -0.000977 -0.001465 -0.001953 ...
 $ sim_m      : num [1:513, 1:30] 0 -0.000488 -0.000977 -0.001465 -0.001953 ...
  ..- attr(*, "dimnames")=List of 2
  .. ..$ : chr [1:513] "1" "2" "3" "4" ...
  .. ..$ : chr [1:30] "sim1" "sim2" "sim3" "sim4" ...
 $ is_residual: logi TRUE
 - attr(*, "class")= chr "curve_set"


$`2`
$`2`$rank_env_NA
Rank envelope test
 p-value of the test: 0.8709677 (ties method: midrank)
 p-interval         : (0.8064516, 0.9354839)

$`2`$random_labelling
curve_set object containing :
List of 4
 $ r          : num [1:513] 0 0.000488 0.000977 0.001465 0.001953 ...
 $ obs        : num [1:513] 0 -0.000488 -0.000977 -0.001465 -0.001953 ...
 $ sim_m      : num [1:513, 1:30] 0 -0.000488 -0.000977 -0.001465 -0.001953 ...
  ..- attr(*, "dimnames")=List of 2
  .. ..$ : chr [1:513] "1" "2" "3" "4" ...
  .. ..$ : chr [1:30] "sim1" "sim2" "sim3" "sim4" ...
 $ is_residual: logi TRUE
 - attr(*, "class")= chr "curve_set"

You can now plot the results from the rank envelope test:

library(ggplot2)
jpeg("rank_envelope.jpg",width = 1200, height = 1200, units = "px",quality = 100)
plot(NA_list_result$"1"$rank_env_NA,use_ggplot2=TRUE)
dev.off()

Nested `for` loop

Fri, 24 Mar 2017 01:00:00 +0100

I already talk about single for loop in a previous post. The same way we did for loop for a single object, we can do it multiple times.

Here a short example: First import the data from here

library(data.table) #load data.table at least V.1.10 (to have fread function)
data_stand <- fread("data_stand_scale.txt") #read the data

data_stand

     Status      mark Stand_ID.m pattern_state
  1: living        NA          1        state1
  2: living       DBH          1        state2
  3:   dead      dead          1        state1
  4:   dead mortality          1        state1
  5: living        NA          2        state3
 ---                                          
188: living       DBH         38        state2
189:   dead      dead         38        state1
190:   dead mortality         38        state3
191: living        NA         39        state1
192: living       DBH         39        state1

And then we can try a nested loop like this:

result_table <- NULL #create a NULL vector
for (i in unique(data_stand[, Status])) {
    new_data <- data_stand[Status == i]
    summary_all <- NULL #create a NULL vector which will be erase at each loop increment

    for (y in unique(new_data[, mark])) {
        summary <- dim(new_data[mark == y][pattern_state == "state1"])[1]
        summary_all <- cbind(summary_all, summary)
        rownames(summary_all) <- i #to have the rownames according to the Status level
    }
    result_table <- rbind(result_table, summary_all)
}
result_table

	summary	summary
living	0	5
dead	32	23

And here are the results. I know that we can do it in a simpler way, it was just here as an example and before presenting you the foreach loop which are used to do parallel computing.

data.table

Wed, 08 Mar 2017 01:00:00 +0100

When you are doing a lot of R, you can find two different kind of people, the dplyr and the data.table people. Indeed, those to packages are used for table manipulation. They are like the ggplot2 function for graphic, when you know them they change your R experience and you cannot go back.

dplyr

I will not talk a lot about dplyr package, because I am not using it. However, you need to know few things about it. Usually, dplyr is used with another package named tidyr, and since recently, you can find one package which will load both of them and more (load also ggplot2, readr, etc.) and named tidyverse.

To be simple, dplyr is very useful to handle object, and specially for sorting, sub-sample, etc. thanks to the new operator %>%. This operator will pipe object to another line to make more change on it. It is very convenient to avoid typing multiple times the object names like this:

set.seed(121)
data <- rnorm(10)
data <- data + 44

# With dplyr package
library(dplyr)
set.seed(121)
data2 <- rnorm(10) %>%
    + 44

identical(data,data2)

TRUE

But, there is a lot of very nice function with dplyr to select, sub.select, sort, replace data from tables.

If you are using emacs, you can define a key-bind for %>%. Here is the lisp code you will need to put in your .emacs config file.

;;Define shortcut key for %>% function
(global-set-key (kbd "<M-f5>") 'dplyr-function)
(defun dplyr-function ()
  (interactive)
  (insert " %>%"))

Thus, in my example, the shortcut will be Meta (Alt) - f5. Of course, you can change it for whatever you like, just be sure that your shortcut is not already use.

You will find a lot of site to learn more of the dplyr package on internet. The user community is large and you will find a lot of question/answer in stackoverflow from the dplyr package.

data.table

I started using data.table a couple of years ago. At first, this is very hard and the syntax is counter-intuitive, but you will use to it. The data.table function is known to be faster than the dplyr package, but the difference of speed will only be consequent for huge data set (millions or rows and hundred of columns).

data.table use less verbose than dplyr but is more complex to write in my opinion. Here is some examples (the data are here):

library(data.table)

data_plot_pat <- fread("data_plot_scale.txt") #read the data
data_plot_pat
dim(data_plot_pat)

1548

setkey(data_plot_pat, Status, mark)#set in the data table "data_plot_pat" first the Status, and then the mark

selected_plot  <- data_plot_pat[Status == "living" & mark == "DBH"] [, .( Stand_ID.m,lng, lat, pattern_state)]#select plot with living status and DBH mark 
# and then subset only the Stand__ID_m name, lng, lat and pattern_state
print(dim(selected_plot))

504

And you can add

selected_plot[,  lat :=  lat + 100 ]#make some change on a existed vector. Here add 100 for each row on the "lat" vector

selected_plot[,  new_colmun :=  "repeated text"]#add a new column with the same value for each row (here this is "repeated text", but it could be numeric as well.
print(dim(selected_plot))

504

As you can see, here there is a new column.

And if you are using emacs and R, I suggest you to add the following package in your .emacs for the function := to have space between it, you can add this code:

;; Electric operator will turn a=10*5+2 into a = 10 * 5 + 2, so let’s enable it for R
(use-package electric-operator
  :ensure t
  :config
  (setq electric-operator-R-named-argument-style 'spaced)
  (add-hook 'ess-mode-hook #'electric-operator-mode))
(electric-operator-add-rules-for-mode 'ess-mode
  (cons ":=" " := "))

Also, the electric-operator will make you code easier to read by adding space between operator such as " == ", or " + ", etc.

Because I am used to data.table and specially love the fread and fwrite function to read and save tables, I did not want to use dplyr. However, if you are a beginner, or you want to read old code easily years later, I suggest you to use dplyr.

Workflow

Tue, 14 Feb 2017 01:00:00 +0100

How you organise yourself when doing R will make it easier for you if you need to go back to your script weeks or years after, and to share with others people.

There is different workflow that you can do. At first, when I was working on a project, I was using a simple folder named analyses and put everything in it with obscure name such as spatial analyses.R or canopy area analyses.R, etc. I was also putting all the outputs (Figures, tables, maps) in the same working directory as the .R scripts.

I think, this make it harder to find again some figures that you did made.

Now I am using since a couple of year another workflow. I first used number before each names to know which script I need to run at first like 1 data upload.R, 2 function point pattern.R and 3 final results.R.

For the outputs, I have .RData saving files in the working directory, but all my Figures are in different folder. Thus, for example, I have a path like this ./living/NA/normal/replicate/Lmm graph/ with every figures of the L_mm functions (a spatial pattern function) for the replicate normal NA (no marks) and living patterns.

I have a little script that help in R for checking if the path is here and make a new folder or not if it already exist.

Here is the little script will check for the path /Project X/Analyses/results RandomForest:

mainDir <- "/Project X/Analyses"#path do the main folder without "/" to the end
  

if (file.exists(file.path(mainDir, "results RandomForest"))) {
                        setwd(file.path(mainDir, "results RandomForest"))
                    } else {
                        dir.create(file.path(mainDir, "results RandomForest"), showWarnings = FALSE,recursive=TRUE)
                        setwd(file.path(mainDir, "results RandomForest"))
                    }

You can even add it in a for loop like this:

mainDir <- "/Project X/Analyses"#path do the main folder without "/" to the end
#you can use here 
# mainDir <- getwd() # to have this script work
setwd(mainDir) 
variables_names <- c("temperature","slope","patterns")
for (i in variables_names) {
    library(ggplot2)
    library(ggthemes)

    if (file.exists(file.path(mainDir, "results RandomForest",i))) {
        setwd(file.path(mainDir, "results RandomForest",i))
    } else {
        dir.create(file.path(mainDir, "results RandomForest",i), showWarnings = FALSE,recursive=TRUE)
        setwd(file.path(mainDir, "results RandomForest",i))
    }
    ## making a empty graph and save it
    ggplot() +
        geom_text() +
        annotate("text", label = paste0("text with variables names  ",i ), x = 2, y = 15, size = 8, fontface = "bold") +
        theme_few() +
        ggsave(paste0("variables", "_", i,".pdf"))

    setwd(mainDir) #here very important to set the mainDir working directory again
}

Here it will check for three different paths in the same subfolder results RandomForest

/Project X/Analyses/results RandomForest/temperature

/Project X/Analyses/results RandomForest/slope

/Project X/Analyses/results RandomForest/patterns

and save in each folder a file names varialbes_temperature.pdf (or slope.pdf or patterns.pdf) in the folder temperature (or slope or patterns).

The outputs figures should look like this:

Creating a function

Sun, 22 Jan 2017 01:00:00 +0100

In R, most of people used build-in functions which are very useful for all basics statistical analyses. When you need to do more complex statistics you can use packages. There is almost every current statistical functions and more in the CRAN repository. However, time to time you will need to build your own function. Usually, when I build a function is because for loop will not be efficient and that I need to put different parameters that I can change. I will take here an easy example that I have previously done for my work. I wanted to build a short function to draw maps and save them if I needed to.

The data

There is two tables in this example. The first one data_plot_scale.txt is data of forest plots location, with different information such as the trees status, the (living or dead), mark (NA for no marks, DBH for trees with diameter information), dead for dead trees position only and mortality for mortality pattern), the stands names (Stand_ID.m), lng for longitude and lat for latitude coordinates and a result like state 1 to 3 in the column pattern_state. The second table is very similar but at the stand scale data_stand_scale.txt.

Those data represent results of spatial patterns (aggregation, repulsion or random) in forest at the plot and stand scales. Because several plots, which could have different patterns states, are in one stands, the stands patterns could be different from the plots scale. Also, because I needed a tree threshold to compute spatial patterns, there is few plots and stands that cannot be analyse for the dead status (stands #4, for example).

library(data.table)
data_plot_pat <- fread("data_plot_scale.txt") #read the data
data_stand_pat <- fread("data_stand_scale.txt")

data_plot_pat

data.table 1.10.0
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
      Status mark Stand_ID.m      lng      lat pattern_state
   1:   dead dead         17 20.11335 49.18592        state2
   2:   dead dead         17 20.11335 49.18592        state2
   3:   dead dead         17 20.11335 49.18592        state2
   4:   dead dead         17 20.11794 49.18627        state2
   5:   dead dead         17 20.11798 49.18874        state2
  ---                                                       
1544: living   NA         33       NA       NA        state2
1545: living   NA         33       NA       NA        state2
1546: living   NA         33       NA       NA        state2
1547: living   NA         33       NA       NA        state1
1548: living   NA         33       NA       NA        state1

When doing functions you will need first to create arguments (vector) that will be used in the function. In this example, I will use status as a proxy for Status (note the S uppercase difference), markk for selecting mark, and then just 2 others FALSE or TRUE vector is I want to have the graphics outputs with the stands and if I want to save same or not in the current directory.

The function

map_draw_blog <- function (status, markk, with.stand, save, ...) { #usually put ... to path others arguments
    library(ggmap)#use library instead of require to stop the function if you did not have the library
    library(ggplot2)
    library(data.table)
     #plot prep data
    setkey(data_plot_pat, Status, mark)
    #below you need to put %in% for the mark to take into account the mark == NA
    for_mapping_plot  <- data_plot_pat[Status == status & mark %in% markk] [, .( Stand_ID.m,lng, lat, pattern_state)] 

    #remove duplicated rows
    for_mapping_plot <- for_mapping_plot [!duplicated(for_mapping_plot )]

     #Stand prep data
    small_data_inf <- data_plot_pat[, .( Stand_ID.m, lng, lat)]

    #compute the mean of lng and lat by Stand_ID.m with removing NA value  only to remove lat and lng NA value
    to_map <-small_data_inf[, lapply(.SD, mean, na.rm = T), by = Stand_ID.m]

    pattern_to_put_map <- data_stand_pat[Status == status & mark %in% markk] [, .(Stand_ID.m, pattern_state)]

    for_mapping <- merge(to_map, pattern_to_put_map, by = "Stand_ID.m", all = T)#all = TRUE to keep all the no match from the 2 tables
    #keep only data with complete cases (remove row with NA values)
    for_mapping <- for_mapping[complete.cases(for_mapping)]
    
    #create the map background using Google satellite data
    google_map <- get_map(location = c(14,46,26,49),  maptype = "satellite", source = "google", zoom = 6)
    if(with.stand == TRUE) {
        if(save == TRUE) {
            ggmap(google_map) +  geom_point(data = for_mapping, aes(
        lng,  lat, color = factor(pattern_state)), 
	size = 10, alpha =.80)+  geom_point(data = for_mapping_plot,
        aes( lng,  lat, color = factor(pattern_state)), size = 2)+  
	geom_label(data = for_mapping, aes(lng,  lat + .2, label =
        Stand_ID.m,color = factor(pattern_state)), size = 4) + 
	ggsave(paste0(status, " ", markk, " map with stand and plots patterns.jpg"))
        } else { #end if save = TRUE
            ggmap(google_map) +  geom_point(data = for_mapping, aes(
        lng,  lat, color = factor(pattern_state)), 
	size = 10, alpha =.80)+  geom_point(data = for_mapping_plot,
        aes( lng,  lat, color = factor(pattern_state)), size = 2) +  
	geom_label(data = for_mapping, aes(lng,  lat + .2, label = Stand_ID.m,color = factor(pattern_state)), size = 4)
        }
    } else { #end if with.stand = TRUE
        if(save == TRUE) {
            ggmap(google_map) +  geom_point(data = for_mapping_plot,
        aes( lng,  lat, color = factor(pattern_state)), size = 2)+  
	geom_label(data = for_mapping_plot, aes(lng,  lat + .2, label
        = Stand_ID.m,color = factor(pattern_state)), size = 4) + 
	ggsave(paste0(status, " ", markk, " map without stand and plots patterns.jpg"))
        } else {
            ggmap(google_map) + geom_point(data = for_mapping_plot, aes( lng,  lat, color = factor(pattern_state)), size = 2)
        }
    }
}

Here you can use the freshly build function map_draw that we just created. The status can be "living" or "dead". For the markk note that you need to put NA without any quote. Also, you can put just F for FALSE or write FALSE, and the same for TRUE, the result will be the same.

map_draw_blog(status = "living", markk = NA, with.stand = T, save = F)
map_draw_blog(status = "living", markk = "DBH", with.stand = FALSE, save = T)
map_draw_blog(status = "dead", markk = "dead", with.stand = T, save = TRUE)
map_draw_blog(status = "dead", markk = "mortality", with.stand = T, save = T)

I edit the graph using Inkscape to put a red arrow on the plots/stands that appear only for living status.

For loop

Fri, 20 Jan 2017 15:31:14 +0100

Using loop with R is not the proper behaviour to have. Indeed, using the apply functions family such as tapply, lapply, and so one are a better way to do it, but for me, this is in some case too complicated. The apply family functions improve the speed rate of T and the management of the memory. Thus, apply functions is relevant to use when you have big data-set to play with. Instead of using apply functions, I used for loop. This is simpler to understand it for me and with moderate data-set (about few thousand of rows and 100 columns) the gain of apply functions is limited. Like this, for loop can be read again quickly even when you comeback to your code years later. I will now provide a small example to understand when you can use for loop and how to used them. Also, note that understanding for loop will lead to use foreach function which allow parallelisation computing. It will be explain in another post.

So, let's begin the for loop example. I will show you how to do spaghetti plot from tree ring data.

About the data

Every year, tree make rings. We can count them to have the tree age, and also measure them to know a little more about the tree and its story. Indeed, tree increment (ring width from a year to another) give information on past climate, past disturbances (pest, drought, fire, ice and wind storms, etc.).

In this data set(tree rings increment) you will find tree rings increment [in mm] from 19 different trees (V1 to V19). The NA value is for year without increment information for this tree (it was not born), and the row #270 is for AD 2011 and the row #1 for AD 1741.

The data look like this:

setwd("./assets/data/") # change following your own working directory
library(data.table) #load data.table at least V.1.10 (to have fread function)
increment_data <- fread("tree rings increment.txt")

##if you did not want to use data.table V1.10 (which allow the fread function use the following code
#increment_data <- read.table("tree rings increment.txt",sep=";",h=T) #for your data,
#Because trees have not the same age, there is a lot of NA value and the length of
#your data frame is equal to the year of the oldest trees (for me it was a tree not show in this sub-sample).
#print(increment_data[1:3])

increment_data[, .(V1,V2,V3)]

data.table 1.10.0
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
         V1     V2    V3
  1:     NA     NA    NA
  2:     NA     NA    NA
  3:     NA     NA    NA
  4:     NA     NA    NA
  5:     NA     NA    NA
 ---                    
266: 128.99 182.54 66.89
267: 130.45 182.80 67.14
268: 131.54 183.24 67.50
269: 132.25 183.48 67.73
270: 132.76 184.34 68.14

We are now going to plot each tree starting from their first year of growth record (i.e. the closest year to the row #1), and then continue until the year AD 2011 (i.e. row #270). Because in data-set there is no year with 0 of increment, we need to add it for each trees using the following code example for one tree:

    # the 3 following code line are just to help to generate the graphic frame
increment_data_unique <- increment_data[,1]
increment_data_unique <- increment_data_unique[complete.cases(increment_data_unique) ]
increment_data_unique <- rbind(list(0),increment_data_unique)
# I am pretty sure that those 3 lines of code are not required.
as.data.table(increment_data_unique)

	    V1
     1:   0.00
     2:   0.71
     3:   1.64
     4:   2.07
     5:   2.46
    ---       
   137: 128.99
   138: 130.45
   139: 131.54
   140: 132.25
   141: 132.76

Here we can see that we remove all NA value, and put a 0.00 value for the first year of growth of the tree

Because we need to do it for each 19 trees, this is easier to do it within a for loop like this:

#pdf("Multiplot radial growth sugar maple between 150 and 160 years old.pdf", width=12, height=8) #to save the plot as PDF

plot(as.numeric(rownames(increment_data_unique))-1,increment_data_unique[[1]],xlab="Age [in years]",
     ylab="radial growth [in mm]",main="Radial growth of 19 trees between 150 and 160 yrs",
     type="n",ylim=c(0,250),xlim=c(0,165)) #generate just the frame of the graphic to have all the same scale

for (i in names(increment_data)) { #for loop to draw one by one, each line of tree increments, starting all in age 0
 #use of names(increment_data) to have the names of each trees from "V1" to "V19"

    increment_data_unique <-increment_data[,i,with=F]#with=FALSE to have the whole column
    increment_data_unique <- as.data.frame(increment_data_unique[complete.cases(increment_data_unique), ]) #remove all the NA value
    increment_data_unique <- rbind(0,increment_data_unique) # add the 0 mm of DBH to start all tree at 0 years-old, 0 DBH
    par(new=TRUE) #allow to combine plots
    plot(as.numeric(rownames(increment_data_unique))-1,increment_data_unique[[1]],xlab="Age [in years]",
         ylab="radial growth [in mm]",
         main="Radial growth of 19 trees between 150 and 160 yrs",type="l",ylim=c(0,250),xlim=c(0,165)) #draw the plots for a single tree.
    #then restart the loop with the next tree, etc.
}
#dev.off() # save the graphic in your directory

If you want, I have put the all code in my GitHub with the data-set Dendro-spaghetti-plot

Convert factor to character

Thu, 12 Jan 2017 01:00:00 +0100

In R, you used sometimes functions to convert vectors to another format with as.character, as.factor or as.numeric.

However, you need to be careful when using the function as.numeric on a factor. Indeed, if the factor contains factor number, it will convert them to the level number and not convert the writing number from your factor.

Here is a short example:

library(data.table)
set.seed(23) #set.seed to have the same random value as in this example
to_convert <- factor(runif(1:10, min = 0, max = 100))

print(levels(to_convert))

[1] "22.3072855733335" "33.1896589370444" "42.3720560967922" "57.6603659661487"
[5] "71.072455169633"  "81.9448956055567" "84.052187949419"  "96.3544549420476"
[9] "97.8130409261212" "99.6611237060279"

data.table(factor_to_convert = to_convert, 
           using_as_numeric_only = as.numeric(to_convert), 
           using_as_character_first = as.numeric(as.character(to_convert)))

factor_to_convert	using_as_numeric_only	using_as_character_first
57.6603659661487	4	57.6603659661487
22.3072855733335	1	22.3072855733335
33.1896589370444	2	33.1896589370444
71.072455169633	5	71.072455169633
81.9448956055567	6	81.9448956055567
42.3720560967922	3	42.3720560967922
96.3544549420476	8	96.3544549420476
97.8130409261212	9	97.8130409261212
84.052187949419	7	84.052187949419
99.6611237060279	10	99.6611237060279

So, be careful when you are using factor with numerical levels and you want to convert them.

Emacs + ESS = loving R

Fri, 30 Dec 2016 01:00:00 +0100

Since my second year of university I have studied Statistic with R on Windows using the basic text editor from the R GUI console. I was happy with it for a couple of years. However, when I start coding more complex and longer function, it was a nightmare to handle human issues, specially parentheses tracking. A professor told me that they were some programs specific to code in R, such as Tinn-R or the well known R-Studio. I also found that Notepad++ as a plugin (nppToR) to link the text editor to R. However, after trying R-Studio and Notepad++, I was not very happy with them. I always did not like the multi-windows design from R-Studio, and Notepad++ was only a better way to use the script function from the R GUI. It was not fully integrated R. Finally, my professor told me about Emacs, and how it was working with R. I first try it with Windows, using the very good Emacs version from Vincent Goulet. It was exactly what I was looking for. A software that can interact with R without a lot of windows, and well integrated with R. Few years later, I jump to Linux using first Manjaro (an ArchLinux distribution), and after Xubuntu to keep the very good XFCE desktop environment.

For the reader who don't know anything about Emacs this is a free software from the GNU licence which can speak R using ESS package (for Emacs Speaks Statistic). But Emacs did not speak only statistic, this is an incredible tool that can speak multiple computers languages, read your email and much more. In fact, I am writing this blog article using Emacs through Org-mode. Emacs is a cross-platform program which can be used with Windows or Mac Os, and also Linux.

I will put more articles in this blog describing how I am using Emacs with R, and for blogging, and hopefully you will consider it to use for your own projects.

Sample Stat

Hide your password in R

Mean of the n last or first

parallelisation computing

Nested for loop

data.table

dplyr

data.table

Workflow

Creating a function

The data

The function

For loop

About the data

Convert factor to character

Emacs + ESS = loving R

Nested `for` loop