Rubyplot project discussion

GSoC GSOC 2018
1

Students:

Please use this forum for discussions regarding the matplotlib project.

2

To kickstart the discussion here I will reconvey the last message that I sent in the mail thread.

Pjotr, I like the Idea of Quick Prototyping over Ruby. I propose that we implement Rendering Layers with properly documented Rendering backends (ImageMagick,GR,QT) along with the Transforms(using CBLAS or Armadillo) in C++ while using Matplotlib as a reference. Meanwhile we can create a Ruby Artist Layer over the existing Renderers in MatPlotLib , creating a reference architecture that can be used for replication of the Artist Layer in C++ . Later-on we can sync them together to create the prototype of the plotting library and replace the Ruby parts with C++. I feel that Transforms (although the part of the Artist Layer) provide one of the most critical functionalities and would require a lot of time and testing to implement. The Affine transforms can be easily implemented in NMatrix and can act as a placeholder while the Transform layer in C++ is implemented.

I am exploring the resources and hyperlinks in the Proposal and making myself comfortable with the low level APIs of matplotlib to get a better grasp of the challenge.

3

Hi Everyone,
First of all thanks a lot for selecting me and Pranav for Gsoc.
I would like to share a common document where we will give a rough outline of our potential design decisions that we will be making for Ruby Matplotlib.
It already has my proposal and a few ideas from Pranav’s proposal.
We will be adding more to it with time.

4

Greetings Everyone!

Before we start planning the Architecture and the deliverables, I think that we first need to decide what the output of the projects should be.
I see three possible products:
a. A Library with just the backend as C++ and other layers in Ruby.
b. A Library completely written in Ruby
c. A Library that is purely written in C++ (with Ruby bridge created later if the time permits).

Before talking about them I would like to talk about matplotlib. The C++ backend source code is confined to the src folder for matplotlib can be found here. It must be noted that the C++ Backend code has been extensively written to create Abstractions and Classes from AGG which are compiled as a .out file during the build and later imported by backend helper scripts. The rest of the backend (renderers which have APIs for python are solely implemented in python only. eg. Cairo backend ). All the other layers have been purely implemented in Python.
In Summary, Entire Matplotlib can be recreated without C++ code, given the correct choice of rendering Libraries.

Now to evaluate the options:

a. A Library with just the backend as C++ and other layers in Ruby: This will be a similar approach to the development of Matplotlib with its backend renderer written in C++ and bindings created to the Ruby core. just like the C++ code in Matplotlib, it will require a lot of helper scripts to handle the pipeline between Python and Ruby.

b. A Library completely written in Ruby: The Backend renderer with ruby bindings can such as RMagick or RCairo can be used along with implementation of the rest of the Matplotlib’s python code in Ruby to create a Ruby based plotting library.

c. A Library that is purely written in C++: This is something which I am deeply interested in. I think that the backend should be implemented in well documented renderers such as ImageMagick and Figure Canvas like QT. After which the majority of the time should be spent on the creation of an Artist Layer in C++. Later Bridges from various languages can be created to various endpoints of the Library.

I would like to know everyone’s thoughts about the same and what the final product(s) should be.

PS: Can the Discourse Administrator please allow new users to post more than two links in a post?

5

@Arafat and @pgtgrly :

Extremely sorry for the delayed response. I had been caught up in University work and there was
little time. However, I will be fully available for the project henceforth.

Based on discussions on discourse and the mailing list, here’s how we should split the project:

  • Arafat:
    • Works on the frontend.
    • Makes a backend-independent frontend layer using mock tests or pymatplotlib for starters.
  • Pranav:
    • Works on the backend.
    • Makes a frontend-independent layer using a similar approach.

We will build this library in pure Ruby for now. The slow parts can be ported to C++ later.
Your projects will be independent of each other during the entire GSOC period. We can work
on integration of backend and frontend after GSOC, since Google mandates independent projects and
there’s nothing we can do about it. Therefore, the interfaces that each of you will create will
only come in touch with each other after 3 months and you need to very careful about the right
design and formalization when you initially propose to make these changes. Good communication
is paramount in this project.

So here’s what you have to do now:

  • Come up with a name for the project. I’ll create a github repo and we’ll start working on it.
  • Come up with two independent proposals: frontend by arafat and backend by Pranav.
  • Put them up on Google docs for all to see and comment.
  • Create milestones and todo lists on GitHub projects to see the project at a glance.
  • While creating your independent projects, you must co-ordinate with each other on the interfaces
    that will eventually bind these projects together.
  • Therefore, create a separate document, specifically for the frontend to backend interface whose
    only job will be document the interface. I’d prefer a Google doc for this first and then port
    to GitHub wiki.
  • This document will be split into two parts:
    • Top level view:
      • This will give a birds-eye view of the interfaces and how they’ll interact. Abstract away
        all details.
    • Detailed view:
      • This will give the nitty gritties of each interface of the birds-eye view.

Also, we must keep in mind that there is another strong linear algebra library in Ruby, called
numo-narray. The narray community is very active and is coming up with several new features
like GPU acceleration for narray. I have used narray for some complex tasks and found it to
be more stable than nmatrix. However, it lacks in documentation, multiple backends and sparse
matrix support. But it has most of the library interfaces that we have in nmatrix like fftw,
gnuplot and lapack and is therefore fully usable in most practical scenarios. Its also
faster than nmatrix for many tasks, and has a somewhat better refined C interface which
can be used by external C libraries.

If we go ahead depending only on nmatrix, it is sure to create more fragmentation in the Ruby
scientific stack. I’d like to avoid this from happening as much as possible. We’re a small
community as it as and more fragmentation will only delay our efforts for making high quality
software.

Therefore, I also want you to think about how you can bridge this gap between nmatrix and
narray by possibly creating a neutral interface that can work with either libraries. For this
interface you can assume that the user will use either only nmatrix or only narray, i.e. there
will no interchange of data between these libraries. Therefore, we can only focus on building
a top-level Ruby interface which can basically be swapped for anything else. I’ve used something
of this sort in daru, where the data can be stored using various Wrapper classes depending
on the container that you want, like ArrayWrapper, NMatrixWrapper or GSLWrapper (links below).

Do post your comments on this forum and lets come up with workable proposals within the next
2-3 weeks so you can begin coding ASAP. If you want to clarify something over a Google hangout
please let me know.

PS: The limit for links for new users has been removed.

PS:
Helpful links:

PS: You should follow @masa16tanaka and @giuse_tweets on twitter for interesting updates.

6

I forgot to mention this in the previous post, but we’ll stick with Arafat’s design as much as possible for the project.

7

Minutes of meeting on 2nd May 2018:

We’ve decided that Arafat will work on a front-end using an existing back-end (Python) and Pranav will work independently on the Ruby back-end.

For this purpose, we will put together a hack where the Python back-end is fed data created by Arafat’s front-end for rendering. Until a Ruby back end is ready, we will create an alternative approach where the Ruby front-end will output data to a file or some intermediate representation that can be picked up by the existing (temporary) back end. This way Arafat can work on his project independently.

Meanwhile, Pranav will work on a backend using an appropriate GUI library (need to choose) and will pick up the data format from the existing python front end. He will need to map out whatever parameters the plotting front end can possibly send to the back end, formalize them and build the back end using these assumptions.

Arafat will be working intensively until the 14th and Pranav will work 7,8,9 due to his exams. Arafat will work on his proposal until the 7th and Pranav will pick up the pace after that.

We will assume an ImageMagick rendering layer for this project.

The next tasks are as follows:

  • Pranav:
    • Finalize an appropriate GUI tool (shoes/QT/GTK).
    • Post pros and cons on mailing list and discourse.
  • Arafat:
    • Come up with a way to ‘split’ the Python matplotlib into front-end and back-end and feed data from the front-end to the back-end without dependency.
    • The splitting can be done by understanding how the data is passed from the front to the back and then inserting a ‘middleman’ that can help in using a python back-end.
    • Arafat will come up with a description of this by end of the week.
8

Supporting both NMatrix and numo-narray is unnecessary. If narray is more stable, it should be the choice. Rarely are people going to need sparse matrix support in a plotting library.

With that said, a lot more people use NMatrix than narray, based on github stars. It’s going to be silly for the library to have to copy NMatrix data storage into NArray storage every time someone wants to plot a thing.

Pick one. Picking both doesn’t avoid fragmentation, it creates more, and makes it harder to support the resulting library.

9

Based on discussions on the mailing list, we’ve decided to name the library ‘rubyplot’. The namespace of the library will look like Rubyplot. We must strive hard to make this library the standard ruby plotting library in the future.

Pranav and Arafat have successfully demarcated their tasks and will now proceed to make a concrete proposal for this purpose.

Here’s the mailing list discussion for this: https://groups.google.com/forum/#!topic/sciruby-dev/2UZgHL-jOjc

10

@Arafat , @pgtgrly:

Any updates on the detailed project plan? Also, you can start using github projects and issues on the repo: https://github.com/sciruby/rubyplot

I also went ahead and registered the gem on rubygems: https://rubygems.org/gems/rubyplot

11

First evaluation useful links and information:

Proposal: https://docs.google.com/document/d/1ZvI3JkRCt4c5BSsYl18Spr33UfCzgTAd8RcM_nqj9mc/edit#heading=h.p1iwk55o0pj3

Pranav’s repo: https://github.com/pgtgrly/GRruby-extension

12

@Arafat:

I needed a clarfication on the Artist layer: when you click an event on the GTK frontend, does it call the same set of Artist methods that it did for the first rendering or is there something else for that purpose?

13

Sorry, but building GTK was something that I am going to be doing at the end so I can’t answer this question right now.
As for something else, I would say that there might be a need to build a few methods for the events API but I will have to figure this one out.

14

Alright. So I’m going to ask Pranav to make a copy of your Artists interfaces and build GR-specific artists using the same API. I think he need not be concerned with the event layer since the GR project is for static plots.

I will shortly communicate with you a way to note common Artist interfaces for both your libraries so that it is easier to club them both together later and differences can be sorted immediately.

15

I am making a document for Pranav to help him figure out the best way to make Artist layer for GR.

Armed with the GR features it would be super easy to do development so this is something I am definitely looking forward to.

I will also make internal changes to my code again just so that we could have as many common functions as possible.

16

Post your observations in this thread instead:

  We'll update your conclusions in the description of the issue so that a reader can immediately see the conclusions and not have to read the whole thing.