# org-table, how to compute Coefficient of Determination between two time fields?

I have two time fields. My productivity and sleep. I would like to run a Coefficient-Determination (As described here: http://www.r-tutor.com/elementary-statistics/simple-linear-regression/coefficient-determination )

Is it possible in org-mode?

E.g

``````|Date | productivity | sleep|
|-----|--------------|------|
|[meh]| 4:00         | 8:00 |
``````
• Voting to close. Not emacs question. But--- load table in your favorite language using Org Babel, convert hh:mm to minutes, and implement the equation... – mankoff Mar 20 '15 at 1:52
• @mankoff Org tables use Calc language (the language of Emacs calculator), so the question has to do with Emacs. Calc can do linear regression and operate on dates, basically, everything which is required to perform what is being asked in this question. – wvxvw Mar 20 '15 at 8:23
• Actually, the problem here appears to be that while Org can reasonably add and subtract times, when it comes to multiplication, the behavior is not what you would expect, so something like `(vvar(\$2)^2)/(vvar(\$3)^2)` will give you absolutely unrelated results (seems like Org translates times using `hours * 3600 + minutes * 60 + seconds` formula and then operates on that, so 4 hours squared gives 129600 hours... – wvxvw Mar 20 '15 at 9:33
• @wvxvw of course it can be done in Emacs. Everything can! Perhaps it is just a poorly written question... The answer is simply "yes", and the OP hasn't explained where they are having trouble. I assumed it was implementing the math. – mankoff Mar 20 '15 at 18:47

My previous answer was just too convoluted (and it didn't actually compute what OP wanted). It appears there's a much simpler way to do it:

# This is actually the way to do it

``````| Date  | sleep | productivity | determination coefficient |
|-------+-------+--------------+---------------------------|
| [meh] |  7.50 |         4.00 |                           |
| [meh] |  8.00 |         5.00 |                           |
| [meh] |  8.50 |         6.00 |                           |
| [meh] |  8.00 |         4.00 |                           |
| [meh] |  8.20 |         5.00 |                           |
| [meh] |  8.70 |         6.00 |                           |
|-------+-------+--------------+---------------------------|
|       |       |              |                0.89847792 |
|       |       |              |                0.89847792 |
#+TBLFM: @9\$4=vcorr(@2\$2..@7\$2, @2\$3..@7\$3)
#+TBLFM: @8\$4=vpcov(@2\$2..@7\$2, @2\$3..@7\$3)/(vpsdev(@2\$2..@7\$2)*vpsdev(@2\$3..@7\$3))
``````

As suggested by the comments. Note that this gives you r, whereas RUserPassingBy's results give r^2 (but, finally, both results are correct).

• Oh my, thank you for your effort here. I'll try to tweak with this. – Leo Ufimtsev Mar 20 '15 at 22:32
• @LeoUfimtsev ok, one thing I already know I did wrong here is that fitted variable should be a vector, not a single value, but I'm not sure how to get that from `fit()`. – wvxvw Mar 21 '15 at 8:06
• One definition of R-squared is cov(XY)^2 / (var(X)*var(Y)). Please add one column \$5 for the R-squared, and try the following. I hope I'm not wrong. #+TBLFM: @8\$2=vvar(@2\$2..@-I\$2)::@8\$3=vvar(@2\$3..@-I\$3)::@8\$4=vcov(@2\$2..@-I\$2,@2\$3..@-I\$3)::@8\$5=(@8\$4^2/(@8\$2*@8\$3)) – RUserPassingBy Jun 25 '15 at 19:03
• @RUserPassingBy yeah... I can't express it how much I hate it when people don't stick to the same nomenclature / use obscure name / barely noticeable diacritics instead of proper variable names. You are totally right, it should be along the line that you wrote, but I think it can be made shorter (there's a function to calculate the mean, so the covariance must be easier to compute). Let me see. – wvxvw Jun 25 '15 at 20:40
• @RUserPassingBy ok, nevermind, I now found the problem. I wasn't counting the last row... – wvxvw Jun 25 '15 at 21:11

if you calculate R^2 by R within org-mode and Babel, it's very simple.

``````#+CAPTION: My productivity and sleep
#+NAME: MY-TABLE
| Date    | sleep | productivity |
|---------+-------+--------------|
| ~[meh]~ |  7.50 |         4.00 |
| ~[meh]~ |  8.00 |         5.00 |
| ~[meh]~ |  8.50 |         6.00 |
| ~[meh]~ |  8.00 |         4.00 |
| ~[meh]~ |  8.20 |         5.00 |
| ~[meh]~ |  8.70 |         6.00 |

#+BEGIN_SRC R :session *R* :results output :exports results :var TMP=MY-TABLE[,1:2]
summary(lm(TMP))\$r.squared
#+END_SRC

#+RESULTS:
:  0.8072626
``````

Here Emacs Calc is!

Org-table is based on Calc function. Calc function vcorr calculates a correlation coefficient among two vectors. Use vcorr with calc-eval in Emacs Lisp with Org-mode.

A Coefficient-Determination is the same as R-square. R-square for a simple regression analysis Y on X equals to a squared value of a correlation coefficient between X and Y.

``````#+CAPTION: My productivity and sleep
#+NAME: MY-TABLE
| Date    | sleep | productivity |
|---------+-------+--------------|
| ~[meh]~ |  7.50 |         4.00 |
| ~[meh]~ |  8.00 |         5.00 |
| ~[meh]~ |  8.50 |         6.00 |
| ~[meh]~ |  8.00 |         4.00 |
| ~[meh]~ |  8.20 |         5.00 |
| ~[meh]~ |  8.70 |         6.00 |
``````

When importing lists from Org-table to Emacs Lisp, you can select rows and columns. :var TMP=MY-TABLE[,1:2] selects for whole rows, with 2nd and 3rd columns.

``````#+BEGIN_SRC emacs-lisp :results value :exports both :var TMP=MY-TABLE[,1:2]
(require 'cl)                             ;; required for (first) and (second) function.
(setq LST (cddr TMP))                     ;; import data to list LST excluding column names and hline.
(setq VEC (apply #'mapcar* #'vector LST)) ;; transpose and change to vector VEC, as Calc accepts vectors only.
(setq X (first VEC))                      ;; X is a vector of sleep ('car').
(setq Y (second VEC))                     ;; Y is a vector or productivity ('cadr').
(setq VALUE-R                             ;; correlation coefficient.
(string-to-number                   ;; calc-eval returns string.
(calc-eval (format "vcorr(%s,%s)" X Y)))) ;; alas, calc-eval accepts only strings.
(setq VALUE-R2 (* VALUE-R VALUE-R))       ;; R-square for a simple regression
#+END_SRC

#+RESULTS:
: 0.8072625698326307
``````

And of course you have the `orgtbl-aggregate` package (available on Melpa). It uses Calc as its backend engine. The `vcorr` function is the one to use, as wvxvw and RUserPassingBy put it.

``````#+BEGIN: aggregate :table "MY-TABLE" :cols "vcorr(sleep,productivity)"
| vcorr(sleep,productivity) |
|---------------------------|
|            0.898477918389 |
#+END:
``````