Final evaluation

About the project:

The Apache Taverna (incubating) suite is consist of Taverna Engine, which handles both Taverna Workbench and the Taverna Server. Taverna Server is used to execute remote workflows and workbench provides GUI for create, run, import workflows. This concept allow scientists who has limited knowledge in computing, limited technical resources to do complex process on Data. Also Taverna engine can be run on standard Windows, Linux or Mac OS.

At the moment Taverna 3 workflows are constructed using SCULF2 API. So, Tavera Engine can’t execute other workflows other than Taverna workflows. This is where CWL comes in ,The Common Workflow Language (CWL). CWL allows one description of a workflow to be run on multiple different platforms as long as they all support CWL.  " CWL builds on technologies such as JSON-LD and Avro for data modeling and Docker for portable runtime environments ".

The project mission is to bring CWL UI support in the Workbench   which is a graphical way to build workflows and one of the steps in Add Common Workflow Language support to Taverna.  

Mentor: Stian Soiland-Reyes
Modules: 

• https://github.com/ThilinaManamgoda/incubator-taverna-common-activities/tree/cwl-licenses/taverna-cwl-activity-ui
• https://github.com/ThilinaManamgoda/incubator-taverna-common-activities/tree/cwl-licenses/taverna-cwl-activity
• https://github.com/ThilinaManamgoda/incubator-taverna-common-activities/tree/cwl-licenses/taverna-cwl-utilities

Org repository: https://github.com/apache/incubator-taverna-common-activities

I had a seperate branch for merging called "cwl-browse": https://github.com/apache/incubator-taverna-common-activities/tree/cwl-browse

My current branch is "cwl-licenses" which is ahead of one commit from the the last branch that got merged "master". 

Work that got merged: 

1. https://github.com/apache/incubator-taverna-common-activities/pull/20
2. https://github.com/apache/incubator-taverna-common-activities/pull/7
3. https://github.com/apache/incubator-taverna-common-activities/pull/4
4. https://github.com/apache/incubator-taverna-common-activities/pull/1

Work that pending for merging: https://github.com/apache/incubator-taverna-common-activities/pull/21

Work completed:

I have implemented CWL UI plugin for Taverna version 2 and 3, but Taverna version 2 plugin is a intermediate step towards the Taverna version 3.

Taverna 3 code repository-commits: https://github.com/ThilinaManamgoda/incubator-taverna-common-activities/commits/cwl-licenses?author=ThilinaManamgoda

Taverna 2 code repository-commits: https://github.com/ThilinaManamgoda/incubator-taverna-common-activities/commits/taverna-2?author=ThilinaManamgoda

Also I have written a tutorial about how to move the UI plugin  from Taverna version 2 to Version 3 in the Apache WIKI. Tutorial: https://cwiki.apache.org/confluence/display/TAVERNADEV/Tutorial%3A+Move+a+plugin+from+Taverna+2+to+Taverna+3

Previous blog posts:

• http://maanadevgsoc2016.blogspot.com/2016/05/abstract-idea-of-technical-stuff.html
• http://maanadevgsoc2016.blogspot.com/2016/06/first-week_5.html
• http://maanadevgsoc2016.blogspot.com/2016/06/adding-service-detail-panel.html
• http://maanadevgsoc2016.blogspot.com/2016/06/new-features-in-service-detail-panel.html
• http://maanadevgsoc2016.blogspot.com/2016/06/new-implementations.html
• http://maanadevgsoc2016.blogspot.com/2016/08/new-way-of-defining-array-in-cwl.html
• http://maanadevgsoc2016.blogspot.com/2016/08/moving-plugin-to-taverna-3_9.html

Experience:

This was a really great project and it has been a pleasure to work with the Apache Taverna Community which is really a great place for a developer to be. Also i must mention my mentor, has been there for me all along. I learned a lot from him and the community. Also i must mention the CWL community as well. They have been really great and patient with me specially Michael R. Crusoe.  I will continue to contribute for the Apache Taverna and CWL community after the GSoC as well.

Moving the plugin to Taverna 3

Introduction

This tutorial is about moving Taverna Plugin from Taverna version 2 to Taverna version 3. In this tutorial, steps towards moving Common Workflow Language (CWL) UI plugin to Taverna 3 is explained. This case has  two main modules:

• Taverna-cwl-activity
• Taverna-cwl-activity-ui

Example service invocation tutorial can be found here which explains how the Taverna 2 activity is implemented.   

Taverna-cwl-activity Module

Activity

Taverna-cwl-activity contains CWLDummyAcivity  class, in Taverna 2 which is responsible for specifying the input and the output ports and doing the actual service that is required. Configuring the Activity is done using CwlActivityConfigurationBean, which holds the required configuration information of the activity.

But in Taverna 3  a JSON NODE object is used instead of CwlActivityConfigurationBean.

Activity class is released from  specifying the input and output ports duty and move that to new class called ActivityFactory, in this case CwlActivityFactory class. When the AcivityFactory class is extended, the following methods should be overridden in such a way that it will return Set containing all the details of inputs or outputs.  

All the details required for configuring the inputs and output are in the configuration JsonNode, Which can be browsed using built in methods such as configuration.get(KEY). Each detail of a parameter(input/output) returned as an  ActivityInputPort or ActivityOutputPort object which can be created like the following:

The edits is an object of org.apache.taverna.workflowmodel.Edits class which is configured in the cwl-activity-context-osgi.xml and cwl-activity-context.xml. //should explained more

cwl-activity-context.xml:

cwl-activity-context-osgi.xml:

The link between  CWLDummyAcivity and CwlActivityFactory is achieved through a URI (Activity Type) which is unique for both classes.

As for dependencies, you can see follow, how the pom.xml is different for two versions

Taverna 2: pom.xml

Taverna 3: pom.xml

As you can see now the packages are named org.apache.taverna.* and previously it was net.sf.taverna.t2.*

Taverna 2:

Taverna 3:

The location and the properties of the repositories are also changed,

Taverna-cwl-activity-UI Module

Example service discovery plugin tutorial can be found here which explains how Taverna 2  discovery plugin is implemented.    

ServiceDescription

In Taverna version 3, it is not needed to provide generics, the type of the Configuration bean for ServiceDescriptions since configuration is a org.apache.taverna.scufl2.api.configurations.Configuration object.

Taverna 2:

Taverna 3:

In Taverna version 3, getActivityClass() method is replaced with the getActivityType()  which returns an URI of the corresponding Activity type.

In this case org.apache.taverna.scufl2.api.configurations.Configuration  object contains JSON NODE, which is the configuration of the CwlDummyActivity and the activity type. Thus in Taverna 3 getActivityConfiguration() method is returning a Configuration object instead of CwlActivityConfiurationBean.

In the ActivityFactory, when the getInputPorts(JsonNode configuration) and the  getOutputPorts(JsonNode configuration) methods are called, the argument  JsonNode configuration is the object, which was  initiated in Configuration object(“c.setJson(cwlConfiguration)”).  

Service Provider

As discussed in the above topics, In service provider also configured using org.apache.taverna.scufl2.api.configurations.Configuration object. In Taverna 2 this is done using Configurable bean which contains necessary details to build a service provider.

Taverna 2:

Taverna 3:

As you can see In Taverna 3 generics are removed since all service provider type are configured using org.apache.taverna.scufl2.api.configurations.Configuration object. In this case service provider is a configurable one. That means user can configure the Service provider inside the workbench.

When the constructor of the service provider is called, Configuration should be passed. In this case this is done by the following method,

In this configuration, user will be asked for a PATH to configure  the service provider.

ServiceIcon

In terms of implementation nothing has changed except In taverna 3 packages are moved to org.apache.taverna.*  from net.sf.taverna.t2.*.

ContextualView

In this plugin HTMLBasedActivityContextualView is used as the extended class for ContextualView class. In that case, details can be represented using HTML tags inside the Service Detail panel. Here you can have a peek what it looks like. In Taverna 3, like previous classes, generics have been removed. Because although  there are different types of Activities, they all have common Configuration type org.apache.taverna.scufl2.api.configurations.Configuration. This object contains JsonNode, which holds required information to build an activity.

Taverna 3:

   Constructor:

      

Taverna 2:        

   Constructor:

As for constructors, in Taverna 3 Activity type object is passed instead of CwlDumyActivity type object. This is because  the type of the activity doesn’t matter as long as the Configuration it holds is valid. Another change is now ColourManager object is passed along with the Activity. This ColourManager object is configured  in “context-osgi.xml” and “context.xml” through the ViewFactory.

Contextual VIew Factory

Taverna 3:

Taverna 2:

As shown above In Taverna 3 generics has been changed. Now it’s org.apache.taverna.scufl2.api.activity.Activity  instead of CwlDumyActivity. This is because now the Contextual View is expecting a Activity type object instead of the Specific Activity type object.

Taverna 3 POM: pom.xml

New features in CWL version 1.0

There are few changes in defining an input or output in CWL command line tool.

• No need to use "-id"  no more. Instead of an input/output can be defined as follow,

Previous	Now

                              

• Type can be defined without the tag "type" as follow,

Previous	Now

• An array can be defined as "<CWLTYPE>[]", for example an Integer array can be defined as follow,

Previous	Now

• Now the type "["null",int]" which means the parameter type can be either a null or an int, can be defined as follow, 

Previous	Now

cwl-utilities module is upgraded for these specifications. Main difference is the output of the YAML parser  is changed. Previously inputs or outputs were mapped to an ArrayList but now it's a LinkedHashMap.

New Implementations

New cwl-utilities module

CWL Dummy Activity has responsibility of reading the CWL configuration and figure out the depth of the parameters of a cwl tool . CWL Contextual View has responsibility of reading CWL configuration and figure out details describing the CWL tool. In order to make those component simple new module has implemented which handles above task for them. Now CWL Dummy Activity and CWL contextual View both can depend taverna-cwl-utilities module to do that work.
Link: https://github.com/ThilinaManamgoda/incubator-taverna-common-activities/tree/stream/taverna-cwl-utilities

Use Java 8 features

Now reading cwl tools taken new approach. Cwl Provider is implemented using new Java 8 features like Lambdas,Streams. Reading CWL tools using SnakeYaml is done trough Parallel Stream which help to improve the performance when there is large number of tools in the given directory.  Since Stream in Java 8 handle the Multi threading for us implementation is lead to cleaner code. Since process is parallel the Arraylist "result" is removed since it can cause concurrency issues, instead now each CWL description is returned as ArrayList it self.

Link: https://github.com/ThilinaManamgoda/incubator-taverna-common-activities/blob/stream/taverna-cwl-activity-ui/src/main/java/org/apache/taverna/cwl/ui/serviceprovider/CwlServiceProvider.java

New Features in Service Detail Panel

New Features in Service Detail Panel

So previously i have showed that how the CWL service detail panel going to be in the Taverna Workbench. Since then i have made improvements like, adding format, label and change the color.

CWL input/ output parameter is consist of  "format" and "label" tags.  Label is a short human readable description of the parameter and it is optional to have. Since service detail panel is represented using HTML, i have  modified the previous implementation  such that panel is defined as a table and all the configuration of the panel is done through the <table> tag.

 Format is a short description of which formats are allowed for the parameter and it's optional to have. Format can be a String, Expression or  referred to an ontology. So if it's an ontology, it should be mapped to  an URI in the Namespaces defined in CWL Tool. 

 

Adding Service detail panel

                         The task is to  implement the Service detail panel for CWL tool in the Taverna workbench. So CWL tool can have a description about it self. But the problem is that the description could be long and couldn't be shown in a single line with the tool name within the available services.
Also each input/output can have own description about it self. This is very useful when designing workflows. So Service details panel is right place for hold this all information. In Taverna this is known as Contextual view and you can find the tutorial here.
                                                              Although this is the basic idea, i am going to approach HTML based solution, in which the descriptions of the tool is formatted using HTML annotation. If you are interested here how it's done. I changed the way how CWL Dummy Activity figure out inputs and outputs, introduced the class "Port Details" which holds the details of each input/output port (depth,description) .  Following pictures demonstrate how CWl tool with descriptions is going to be appeared in the Taverna Workbench, although actually it isn't the best look, but i am working on it. My implementation can be found here.

First week

In this week i have tried to implement the part which will read the cwl file and add input ports in the CWLActivicty. This is done inside these two modules,

• taverna-cwl-activity-ui
• taverna-cwl-activity

taverna-cwl-activity module contains the CWL Activity and the CWL Activity Configuration classes. Inside CWL activity, method called "configure" will go through the CWL configuration and add input ports accordingly. I have also modified the example which is in the Tutorial Service discovery plugin    to demonstrate how CWL tools will appear on the Taverna workbench. Inside CWL service provider it will go through the directory which is user entered , and read CWl tool using SnakeYaml library and create CWl configurations. 

This is the exampletool which is provided by taverna and i modified it to show a peak of what i am doing !!

Give the location of the CWL tools

Services are imported

Select the tool and drag into the work space

Right click and select show ports 

Final View