GSoC 2108 Project [ Advance features in daru-view ] discussion

[Prakriti]

Report for 22nd May:
I spent the day working on custom styling CSS.
There are two modes of Highcharts: Classic and Styled.
Classic mode provides some custom CSS styling and does not require any additional dependencies while styled mode uses js dependency (like https://code.highcharts.com/stock/js/highstock.js) replacing the older one + the CSS dependency and provides full custom CSS styling.
I have implemented the Styled mode for web frameworks but as there is no method to load CSS files in IRuby notebook, so I have kept classic mode for IRuby notebook where some styling can be provided.
User needs to provide the required CSS styling in CSS option.
I have opened a PR regarding the same.
Difficulties: For loading CSS in IRuby notebook, custom.css could be a rescue but then everytime user needs to load the CSS files in custom.css. I am searching for some other way out (refer).

[Prakriti]

Report for 23rd May:

1. I added examples of the custom styling CSS in IRuby notebook in this PR.
2. I worked on separating the dependent js files. User can provide the required js files in modules option as a list of Strings. In IRuby notebook, these will be loaded when show_in_iruby is called (working fine now, had some problem earlier). I have implemented it in this PR. I have modified the examples of HighMaps and HighStock accordingly.
   In web frameworks, div method is called. So I tried loading the dependent js in to_html method which is working correctly. My concern is that the dependent files of the modules are not loaded in the head tag instead they are loaded into the body before the script of the chart, so will it be a problem?
3. Added HighMap example for data as Daru::DataFrame in this commit.

Also, have a look at the revised blog post for the first week.

[Prakriti]

Minutes of the meeting (26th May’ 2018) :

1. HighMaps: We need to add more examples of HighMaps with data in Daru::DataFrame or Daru::Vector format. All the examples of HighMaps can be found here.
2. Transfer of all the js and CSS dependencies to the assets folder.
3. The folder structure of the js dependencies should be maintained.
4. All the constants should be kept in a separate file.

[shekharrajak]

@Prakriti, write down your proposed ideas on GoogleCharts, your current research, what do you want to implement and what not.

We recommend you to explore more about googlecharts from official site and let us know your research.

[Prakriti]

Minutes of the meeting (3rd June’ 2018):

1. GoogleCharts: We would be implementing the following in GoogleCharts:
   a. ChartWrapper class that wraps around the chart and does all the loading and data source querying.
   b. ChartEditor class that provides runtime editing.
   c. Exporting of GoogleCharts to different formats.
   d. Multiple Charts in a single row (Needs to be done for all adapters).
   e. Importing data from google spreadsheets. (Have already opened a PR. Facing some problems earlier but now fixed).
   f. Adding Toolbar in GoogleCharts to export the data in CSV and HTML format(Optional).
   g. Adding listeners in GoogleCharts (Optional).
   h. Adding more methods of DataTables (Optional - will be doing only if we get some solid use case).

2. 3rd parameter: Will be adding 3rd optional parameter that will contain the hash of options created by us like modules, chart_class and CSS in HighCharts. This will be done in a separate PR.

3. Updating Daru-view wiki: Wiki needs to be updated with the highcharts features added, with examples and the future possibilities in highcharts.

[shekharrajak]

Hello @Prakriti,

Please let us know the update and fill this wiki page, mentioning what is completed till phase 1, TODO part (challenges you faced), and next plan. You will see the edit button for wiki page once it is public.

Regards,
Shekhar

[Prakriti]

Sure, I will update the wiki page.

[Prakriti]

Update on the work in daru-view:
For the past two weeks, I worked on the following tasks:

1. Exporting of HighCharts in different formats: Through this feature, user can export the HighChart in various formats (PDF, PNG, JPEG, and SVG) using the code. It is running fine in web frameworks. Progress can be tracked via this PR.
2. ChartWrapper: It facilitates the use of view option and paves the way to implement ChartEditor. I have opened a PR to implement this feature.
3. Import data from Google spreadsheet in google charts: I opened this PR earlier but was facing some issues which have been resolved now.
4. ChartEditor: Right now I am working on it which provides the facility to edit the Chart on the fly. Check out this PR for its implementation.

Challenges and difficulties:

1. Exporting of HighChart in IRuby notebook can be done either offline and online and both have their pros and cons. I am still confused which one should be implemented. Right now, I have kept the online exporting.
2. I am facing some problem in the display of ChartEditor dialog box in IRuby notebook. I tried a lot regarding the same but am still facing the issue.
3. In GoogleCharts, js script of the Chart is generated in two ways, i.e., with script tag and without script tag while we need only the latter one.

[Prakriti]

Summary of the meeting (12th June):

1. ChartEditor: There is a bug that Dialog box of chart editor is appearing behind the IRuby notebook. Still trying some way to figure this out.
2. Exporting of HighCharts: Keeping the online exporting in IRuby notebook and offline in web frameworks. Also need to try this in console.
3. Importing data from Google spreadsheet: Need to add more tests to check the validation of URL. Further, there might be some way to retrieve the data from the URL as opening the link provides the JSON data.
4. Rails examples: More examples needs to be added. There should be three types: simple, intermediate and advanced.

[Prakriti]

Update on daru-view:
This week I worked on Charteditor and displaying of multiple charts in googlecharts.

1. Displaying of multiple charts in a single cell in IRuby notebook and in a single row in web frameworks has been implemented and the PR is open for review along with the Chartwrapper PR. Check out the examples here.
2. ChartEditor: It is implemented in this PR (must be reviewed after Chartwrapper gets merged) and is working fine in web framewroks but in IRuby notebook it appears behind the elements of notebook like this:
   I tried a lot to figure out this problem but it still persists. The div of this dialog box is generated dynamically when we click on the edit button and it remains until we click on the ok or cancel button.
   What I tried:
   a. Increasing the z-index of dialog box in developer’s tools which worked well, so increasing the z-index of the dialog box is a solution.
   b. Increasing the z-index of the dialog in the chart_div using class name(as dialog box does not have id).
   c. Incresing the z-index of the dialog in the button click listener.
   d. Creating async function in listener and then increasing the z-index (as I was unable to get the dialog so I used sleep in it).
   e. Reducing the z-index of other charts.
   Nothing worked out. I was unable to get the dialog box element through the script as when the dialog box is generated it makes aria-hidden="true" for other divs.

[Prakriti]

Minutes of the meeting (23rd June):

1. Exporting of GoogleCharts: GoogleCharts provides the chart.getImageURI() method which in turn provides the access to a PNG image of a chart only. This has been implemented but for more types like pdf, jpeg and png, there are other solutions. For pdf specific, we can use jspdf which is working fine in rails app but again is producing an error in IRuby notebook as the jspdf dependency is not loaded properly in IRuby notebook. I tried it in this PR. (See example)
   Another solution is using cloudformatter (examples) but again this does not work in IRuby notebook. It provides exporting options to jpg, png and pdf. I tried it in this PR.
   Right now, I am keeping the png format only as it works for both.
2. Handling events (adding listener) in googlecharts: JS will be provided in the string format in the third parameter or as plot.add_listener(event, callback).
3. Multiple Charts in a row: Need to create a separate class as Plot class is for creating a single plot.
4. There is an issue that whenever we update the jsapi in googlecharts, it causes an error.

[Prakriti]

Update on daru-view:
For the last few days I’ve been working on the following tasks:

1. Exporting of GoogleCharts: I’ve implemented the export to PNG in googlecharts (for corecharts and geocharts). GoogleCharts official does not provide any method to export the chart and some other methods are not working properly with the IRuby notebook. I have opened this PR for the PNG implementation.
   Further, I’ve kept this PR for the future reference. It uses jspdf (for exporting to PDF) and cloudformatter (for exporting to pdf, png and jpeg).

2. Event Handling: With this feature, a user can apply multiple listeners to a google chart/table to handle various events. I have opened this PR for its implementation.
   Event handling can be done in Highcharts easily through the highcharts options like this, so in googlecharts also, we are handling events through options (in third parameter).

3. ToolBar: I was going through the js code of toolbar to incorporate it in daru-view. However, in the documentation, it is mentioned,

   To use a toolbar, your visualization must get its data from a URL; you cannot pass in hand-populated DataTable or DataView objects. You will pass the URL of the data used to populate your visualization into the drawToolbar() method.

   So, I think it would be of less use as our primary aim is to visualise the data from Daru::DataFrame and Daru::Vector.

[Prakriti]

I was digging more into the Google Charts website and learnt about the formatters in Google DataTables. Here is the document that describes how the API might look in daru-view. @shekharrajak can you please go through it once and also will this feature be good to have in daru-view?

[Prakriti]

Minutes of the meeting (1st July):

1. Multiple Charts: Plot class should inherit PlotList and each plot object in the list should work according to its adapter. Thus, each plot object should call its own div.
2. Export GoogleChart/ HighChart in terminal: to export the chart in terminal using export method, we can export the HTML file to download the chart and then will open it in browser through code. This will generate the popup to download the chart. We need to find a way to distinguish whether we are running code in console or web frameworks (for rails we can use if defined? Rails)
3. Formatters: There is already a class created for formatters in googlevisualr. We can use that class to add this feature in daru-view.
4. Chartwrapper: Need to create a separate module and use it for both table and chart class.

[Prakriti]

Minutes of the meeting (10th July):

1. Another method Daru::View.init_ruby should be created to load the libraries (right now, at object level init_iruby can be used).
2. Need to add more examples of handling events in google charts.
3. Add more examples and screenshots of examples in demo_daru-view.
4. Styling should be done in rails examples.
5. Name of daru-data_tables should be changed to daru-view_datatables.
6. Proper documentation and testing needs to be done in daru-data_tables.
7. Fill up the progress report for second phase.

[Prakriti]

Update on daru-view:
For the past few days, I have been working on the DataTables (PR #104 in daru-view and PR #20 in daru-data_tables) and have achieved the following in it:

1. Use javascript Array rather than HTML table in DataTables: Right now, data is converted in the form of HTML table. I have shifted it to javascript array for the further server-side implementation (as datatable can take Array of Array).
2. Resolved this issue in DataTables.
3. Loading large set of data piece by piece: I have implemented server-side processing (using ajax option in DataTables) in DataTables for more than 50000 data rows and client-side processing below that.
4. Updated the API of daru-data_tables: I have updated and refactored the API of daru-datatables.
5. Added tests and documentation: I have added specs for the daru-data_tables gem and datatables.rb (in daru-view) along with the documentation.

Apart from these, I added some rails examples in demo_daru-view, committed on some old PRs and added load_libs_in_iruby method in this PR to load dependencies of multiple adapters in IRuby at once.

[Prakriti]

Minutes of the meeting (21st July):

1. Add datatables examples in rails in demo_daru-view.
2. Add a constants file in daru-data_tables also.
3. Create another method dependent_scripts which will call load_libs_in_iruby (similar thing for dependent_script).
4. Along with rails examples, add sinatra and nanoc examples too in demo_daru-view.
5. Write a description of each PR in the comments along with the examples.
6. Add rails examples in daru-data_tables.

[Prakriti]

Minutes of the meeting (31st July):

1. Need to create separate files for separate modules (GoogleVisualr, Display, DisplayJavaScript).
2. Add real-world examples in daru-view_io_rails repo.
3. Remove a bunch of lines in the head tag. Understand the internal working of rails (railtie).
4. Change Daru::DataTables::DataTable to Daru::View::DataTable in daru-data_tables.

[Prakriti]

Minutes of the meeting (4th Aug):

1. The landing page of demo_daru-view must be improved.
2. Include real-world examples in daru_examples_io_view.
3. Implement 3rd parameter in HighCharts (chart_class, CSS, modules).
4. While renaming the DataTables, init_script should be called using DataTables.
5. Include rails examples where dependent JS are included from application.js file in demo_daru-view.

[Prakriti]

Update on daru-view:

1. Formatters in GoogleCharts (PR #110):
   The Google Visualization API provides formatters that can be used to reformat data in a visualization.
   For more details, visit this page.

2. Added a method to load dependent scripts for multiple adapters (PR #107):
   Have a look at these examples.

3. Rake Task to add new adapter templates (PR #112):
   Added a rake task to generate the sample template for the adapter.

4. Rename Daru::DataTables::DataTable to Daru::View::DataTable (PR #114 in daru-view, PR in daru-data_tables):
   As daru-data_tables is a plugin gem for daru-view, it needs to be initialized in a different way. Now, it can be instantiated as Daru::View::DataTable.new(data, options).

5. Reduce a bunch of lines due to JS files in source HTML in rails (PR #115 in daru-view, PR #23 in daru-data_tables):
   This aims to remove a bunch of lines that occur in the head tag in rails. In rails, the one line of code we actually need to write in daru-view is for a class called Engine, this is going to inherit from Rails::Engine, and this is more or less saying that when rails loads this gem, it’s going to look for this Engine class, and if it’s defined, it’s going to treat this gem’s directory structure a little bit differently. It’s going to work sort of like your rails application, almost as if you had a rails app inside of a rails app.
   To carry this out, dependent JS and CSS files are shifted to vendor/assets/javascripts and vendor/assets/stylesheets respectively, so that, JS can be included in the app/assets/javascripts/application.js file as:

//= require highcharts/highcharts                                                           
//= require highcharts/highcharts-more
//= require highcharts/map
//= require jquery-latest.min
//= require jquery.dataTables

and CSS files can be included as:

 *= require jquery.dataTables

Include the below line in the head of the layout file:

<%= javascript_include_tag "application" %>
<%= stylesheet_link_tag "application" %>

Note: The previous method to use <%=raw Daru::View.dependent_script(:highcharts) %> will also work.

Difficulties:
Right now this has only been done for rails. Other web frameworks do not provide much flexibility and there are fewer resources available to search. I am trying to reduce lines in Sinatra and nanoc too but still facing many difficulties. Loading JS and CSS files in the app (by keeping these files in the app) can be done, but loading these files from daru-view in the sinatra/nanoc/other frameworks is not an easy task. I have also opened this issue in the sinatra repo.

6. Improved Code Quality (PR #116, PR #76, PR #73, PR #74):
   Improved code quality of the project by fixing various rubocop comments in these PRs and also during working in other PRs.

7. Added examples in web frameworks:
   Rails examples:

• Importing data from google spreadsheet in google charts - #2
• HighCharts, HighStock, HighMaps examples - #3
• Custom CSS in highcharts example - #4
• ChartWrapper examples - #6
• Multiple Charts using PlotList examples - #8
• DataTables examples - #10
• ChartEditor examples - #9
• Formatters examples - #11
• Handling events examples - #12

Sinatra Examples:

• Importing data from google spreadsheet in google charts - #5 commit
• HighCharts, HighStock, HighMaps examples - #14
• Custom CSS in highcharts examples - #13
• ChartWrapper examples - #5 commit
• Multiple Charts using PlotList examples - #5 commit
• DataTables examples - #5 commit
• Handling events examples - #5 commit
• Formatters examples - #16
• ChartEditor examples - #17