Project Title: Migrate RESTful web services from Struts to Spring

Primary Mentor: Daniela Butano

Backup Mentor: Julie Sullivan

Student: Prabodh Kotasthane

Project idea here.

Source Code and Downloads

• Source Code for InterMine Web-services Spring Boot Module : InterMine Web Services
• Source Code for Bio Web-services : Bio Web Services
• Swagger Specifications for InterMine and Bio Web-services : Swagger Specifications

Major Objectives

• Migrate the web-services from Struts (Http servlet-service architecture) to Spring
• Well documented APIs using OpenAPI Specifications
• Functional and unit testing

Extra Credit

• Add the APIs to SmartAPI project

Project Overview

InterMine integrates biological data sources, making it easy to query and analyse data. Presently InterMine uses Struts framework which is outdated. InterMine provides RESTful web-services which facilitates to execute custom or templated queries, search keywords, manage lists, discover metadata, perform enrichment statistics and manage user profiles. The main objective of this project was to migrate the web-services from Struts (Http service-service architecture) to Spring framework and document the APIs with Swagger in compliance with OpenAPI Specifications. OpenAPI specifications are easy to write and Swagger Codegen, which supports Spring, makes the job of developer easy by generating the code stubs which can be modified to render the services. Also Swagger UI provides an interactive documentation which also provides the test calls facility to quickly test-run APIs in your browser only. InterMine currently contains the web application and the web-services bundled under a single module “webapp”. InterMine team is now able to split the webapp module into three modules namely webapp, webservices and webcore. The migration of the webapp was not the part of this project.

Migrate the web-services from Struts (Http servlet-service architecture) to Spring

• Previously the webapp module contained both, the InterMine web application and the web-services. The pre-requirement for this project was to split the webapp into three modules namely webapp, webservices and webcore. This pre-requirement was successfully fulfilled by InterMine team and we had independent webservices module which specifically provide the InterMine web-services. The Implementation plan to migrate the web-services is as follows:
  • The mappings of all the web-services to their respective Servlets were present in the Web.xml file of the webservices module. These mappings also defined and described the metadata about the web-service. I used this data to write Swagger/OpenAPI Specifications.
  • I used Swagger Codegen to generate code stubs. Various API interfaces, controllers, model classes and integrated test classes were generated.
  • I migrated the previous struts based webservices module to a Spring Boot module. Then integrated the code stubs generated above with this project ensuring that there are no errors.
  • Next I migrated the business logic for the API. This was, according to me, the trickiest part of the migration flow. The service classes were then changed to return the model objects which eliminited the need of many classes such as Output and Writer classes.
  • Finally we can deploy the web services and test them live on the swagger-ui. The above steps were followed for all the webservices to migrate them.

Well documented APIs using OpenAPI Specifications

• The documentation of all the APIs written using the OpenAPI specifications lies within the code, linked to the API interfaces. Swagger helps us in doing this. User can easily have a look at the web service documentation through the Swagger UI. Also, the OpenAP spacifications for both, InterMine and Bio webservices, is present in respective files.

Functional and unit testing

• Functional testing can be done with the help of swagger-ui. All the web services are tested and running fine. However, unit testing of web services is still pending due to lack of time as I felt there were a bit too much of web services. This would probably be done soon.

Add the APIs to SmartAPI project

• This was a bonus deliverable, which was supposed to be done if some extra time is left. Unfortunately, due to lack of time, some discussions are still pending regarding this. Hopefully, I will be able to complete this task soon.

This was really an awesome project to work upon. This project included development of large API layer consisting of a lot of different types of functional endpoints, which challenged my coding and time management skills. Also, I learnt many new technologies, like Swagger and Spring Boot, which was challenging and fun part too.

Contributions

All the commits:

• Added webservices-spring module.
• Added test model service.
• Updated build.gradle and Removed struts-config.xml and web.xml .
• Shifted to webservice Project.
• Migrated /model endpoint.
• Migrated /version service. Removed webservices-spring project. Removed agentIsRobot() and checkEnabled(). Removed HttpServletResponse dependency.
• Migrated /summaryfields service.
• Migrated /classkeys and /branding services.
• Migrated /schema service.
• Migrated /facet-list, /session and /web-properties services.
• Migrated /templates/system and /semantic-markup/datacatalog services.
• Migrated /facets and /users service.
• Setting up correct response headers and loading the web.properties properly.
• Migrated /semantic-markup/dataset, /semantic-markup/bioentity, /permanent-url, /sequence, /query/code, /search and /data/{type}. Changes related to setting up correct headers.
• Issue of 406 Error resolved.
• Code review changes. Removed unwanted code.
• Refactored models. Added JSONModel and InterMineController. Moved setFooter to InterMineController.
• Removed setFooter() and sendError() dependecy from WebServiceSpring class.
• Moved setHeaders to InterMineController and injected format into to the services.
• Migrated /user/whoami service.
• Migrated /user/token endpoint.
• Added initController() method.
• Solved swagger authorization issue. Migrated /templates endpoint.
• Migrated /template/upload service.
• Migrated /user/deregistration/service.
• Migrated /user service.
• Migrated /user/preferences service.
• Migrated /user/tokens, /user/tokens/{uid}, /templates/{name} and /template/tags services.
• Migrated /lists, /lists/append, /lists/rename and /list/tags endpoints.
• Migrated /lists/union, /lists/subtract, /lists/intersect and /lists/difference endpoints.
• Migrated /widgets, /queries, /query/results, /listswithobjects and /lists/jaccard-index endpoints.
• Migrated /lists/shares and /lists/invitations endpoints.
• Migrated /user/queries/{name}, /user/queries, /query/upload, /ids/* and /path/values endpoints.
• Migrated /query/tolist, /query/append/tolist, /template/results, /template/tolist and /template/append/tolist endpoints.
• Added tags for better grouping of web services.
• Migrated /list/chart, /list/table and /list/enrichment endpoints.
• Changes related to some output format issues.
• Support for bio webservices. Bio webservices swagger added.
• Migrated all bio webservices.
• Migrated JBrowse endpoints.

Resources and Demo Presentations

Community presentation for GSoC 2019.

My presentation slides.

Initial blog post on GSoC 2019.

Thoughts on GSoC

This past summer was again a great experience for me following the last edition of GSoC. Google Summer of Code with InterMine was an amaing coding and time management experience for me. InterMine community is awesome. People here are always excited and available to help. My mentors did help me promptly whenever needed. This edition of GSoC tested my time management and coding skills as the project was a bit lengthy one. I had new challenges every week which were needed to be resolved quickly. GSoC 2019 agian gave me the exposure to work with a big code base and various different technologies which was really a great experience. Overall, GSoC with InterMine made me more confident and helped me in developing technical, communication and managerial skills. Kudos to whole InterMine community and GSoC!