Roadmap

From ETaxonomy

Jump to: navigation, search

'

Project Charter Project Roadmap Requirements ApplePie

Contents

Public Roadmap

To be filled in here then moved to another page when we are happy with them.

SPNHC DEMO WEEK OF 2012-JUN-10

"Target for SPNHC: (i) collecting annotations from a client; (ii) transporting annotations through an FP network that identifies recipients; (iii) the recipients ingest the annotations into their databases 2nd wk of June 2012 Consenus on this target." (2011Oct25)

  • Demonstration at SPNHC 2012 (June 2012).
    • Target Use Case: Annotate Specimen
    • Make new determination annotation in Morphbank(and/or Symbiota).
    • Transport annotation through a FilteredPush network instance consisting of at least three network nodes.
    • View annotation in Mapper UI at instance of Specify6 database.
    • Accept and ingest annotation into Specify 6 database, with resulting action annotation sent to network.
    • View, in web client to network, base harvested record plus new determination annotation plus action annotation recording acceptance of new determination.
  • Early Fall 2012
    • Target Use Case: Remote Data Capture
    • Transcribe new specimen record from label images in Morphbank.
    • Transport new specimen record annotation FilteredPush network instance.
    • View annotation in Mapper UI at instance of Specify6 database.
    • Accept and ingest new specimen record annotation into Specify 6 database, with resulting action annotation sent to network.
    • Implement automated ingest of authorized new specimen record annotations.
    • Define a quality control test for harvested data, implement as a kepler workflow embedded in a network analytical capability.
    • Enable end user authentication through DataOne services.
    • Network architecture uses configurable pipes rather than static node configuration.
  • End of 2012
    • Target Use Case: Find Duplicates
    • Implement find duplicates use case within Specify6.
    • Target Use Case: Quality Control New Record
    • Implement potential duplicate detection as Kepler workflow embedded in network.

Related

TASK LIST IN MANTIS ("Issue tracking: SourceForge MantisBT." 2011Nov08)

Diagram of Network, Components, Capabilities, Clients, and Broad Buildout Plan
Components of ApplePie instance of FilteredPush Network


Key Milestones

Organized by targets they support.

Demonstration at SPNHC 2012 (June 2012)

Target Use Case: Annotate_Specimen

  1. FP Access point able to accept encrypted, signed FP Messages, return a message ID, pass the message to Triage, job planner able to create job plan, triage able to pass job plan to job runner.
  2. Job runner able to locate network capabilities for executing steps in job plan from static network configuration.
  3. Annotation store able to accept annotations from a a job runner.
  4. Message distribution system able to deliver an annotation message to an interest based on values found in the message.
  5. Message distribution system able to deliver an annotation message to an interest based on values not present in the message.
  6. FP network able to accept subscribe to query messages, create corresponding interests, and allow clients to poll those interests.
  7. Annotation Processor able to retrieve an annotation message from an interest, display it to a user, and ingest the content of the annotation into a Specify6 instance. Requirement3
  8. Annotation Processor able to retrieve an annotation message from an interest, display it to a user, and ingest the content of the annotation into a Specify6 instance, and generate corresponding response annotation.

Project complete 2013-Apr-01

Key milestones

  • Transcription of label data into new specimen records within Morphbank, transport of new specimen record annotations for ingest into Specify6.
  • Kepler engine able to function as FP Network analytical capability.
  • Dynamically configurable network nodes

Current Status

Note: Edit the current status to reflect the current status of each component.

Open Issues

Closed Issues

The code behind the demonstration is currently only a working outline of an all-in-memory network environment with data providers, indexes ("global knowledge"), a pub-sub service, and a client that manages subscriptions for users registered with the client. The demonstration is a worked example that logs events to the console and can be run with JUnit.

  • Dev Environment: Ant build files with tasks for compiling and running unit tests. All projects now using Maven for dependency management.
  • Documentation
  • Clients
    • Annotation Processor:
      • can parse/validate the annotations through the predefined sparql quereis
      • can communicate with the messaging system of the FPush network to get interested annotation and send out response annotation
      • working on web interface for the SPNHC demo.
      • Drivers: Specify6 Driver. OAuth provider servlet verifies Specify user credentials over http. Can retrieve data.
    • Client Helper: Can generate annotation RDF for new/updated annotations from a Specify Database. Can generate four types of apple pie annotations (cluster duplicates, new determination, update determination, response annotation). The client helper can now also be configured to support any rdf/xml document generated from java classes. Also exposed as a web service for use by clients (Symbiota, Morphbank) See: DeterminationToRDF Tool and Rdf_Handler
    • MorphBank: Can make local installation of Morphbank. See: Installing_Morphbank. Morphbank can now inject an annotation into the network (using the client helper webservice) by creating a new annotation on an image or by editing an existing determination.
    • Symbiota: Can build, install and run Symbiota See: Installing_Symbiota. Symbiota can now inject an annotation into the network via the add a new determination form.
    • Web Client: Integrated Annotation processor and reworked prototype web client implementation as UI for Annotation Processor.
    • Specify: Can build, install, and run Specify with test data. See: Installing_Specify
    • Kepler Client: Kepler Kuration package.
  • Network
    • Access Point/Triage/Job Planner/Job Runner: Code generated from design documents and modified. Working single node Tomcat servlet implementation of a FP Access Point web service over Triage/JobPlanner/JobRunner. Working single node Glassfish container implementation of a FP Access Point web service over EJB Triage/JobPlanner/JobRunner. In memory stub. Migrated from NetBeans project to Eclipse. Configured for deployment on firuta or local machine using maven.
      • Implemented jobs for message types:
        • FPMessageType=PING, Job=PingJob (Needs change to NETWORK_STATUS), ping should just produce a PONG from the access point, a job.
        • FPMessageType=QUERY, Job=ApplePieSparqlQueryJob
        • FPMessageType=ANNOTATION, Job=ApplePieAnnotationJob
        • FPMessageType=FP_FIND_SETS, No implementation yet. Job=ErrorJob
        • ApplePieMessageType=NOTIFICATION, No implementation yet. Job=ErrorJob
        • ApplePieMessageType=REGISTER_INTEREST, Job=ApplePieRegisterInterestJob
        • ApplePieMessageType=DELETE_INTEREST, No implementation yet. Job=ErrorJob
        • ApplePieMessageType=FIND_DUPLICATES, No implementation yet. Job=ErrorJob
        • ApplePieMessageType=RUN_ANALYSIS, No implementation yet. Job=ErrorJob
        • ApplePieMessageType=ADD_ANALYSIS, No implementation yet. Job=ErrorJob
    • Network Transport Layer = Overlay Network: Nothing yet.
    • Network Capabilities
      • Annotation Store: Redstore installation See: RedStore_Installation_on_firuta. Not yet connected to Triage implementations. Currently using Jena triplestore apis (SDB) over MySQL to store incoming annotations as triples. Used by the access point via the AnnotationJob.
      • Harvesting and network data cache
        • Data Providers: Some work on Specify6 OAI-PMH service.
        • Harvester: Nothing yet.
        • Network Cache & Index: Stub
      • PubSub: Minimal JMS implementation. Mimimal Store and Forward messaging implemented with MySQL.
      • Analysis: Nothing yet. Needs further definition.
      • Generic Mapper: Not under current development. Work focusing on Annotation Processor. Explored writing a query rewriting mapper and a less generic configurable mapper. See http://filteredpush.svn.sourceforge.net/viewvc/filteredpush/FP-Drivers/FP-SQLDriver/branches/mapperPOC/

Target Dates

 

2012-Jan-10

The demonstration can be run as a one-servlet web application. A web user can register themself, search the index, retrieve a record from the index, subscribe to a query, publish an annotation, and receive a notification. done

Demonstrated at: 2012Jan18

2012-Feb-10

  • Dev Environment: Dependency management file (limited ant builds). Components up and running but not necessarily in communication: Specify client Issue 11 closed, Symbiota Issue 10 closed, MorphBank Issue 8 closed, Web Client done. Network nodes instantiatable only in memory.
  • Documentation
    • UML Model for Network Revised to reflect AOD annotation. not done
    • FP Client API Definition FP Annotation message defined probably done, see uml model
    • Code some Javadoc done
    • Deployment instructions Developer instructions for setting up MorphBank and the index, and for populating Specify, MorphBank, and the index with data. Developer instructions for setting up a network with a client. done except for index and network
    • AOD Manuscript submitted. not done yet
  • Clients
    • Annotation Processor: rough draft Web UI for comparing annotation with records of the same type. Requirement9 Test set of >10k new determination annotations generated from HUH-Specify data. Testing on Update Determination and Insert Determination.
    • Web Client: Integrated as UI for annotation processor. done JSF client has UI forms for user registration, login/logout, search, record display, subscription list display, subscription notification list display, and notification display, updated from prototype. done not all forms have backing functionality working yet
    • Client Helper: Can generate annotation RDF for new/updated annotations from a Specify Database. Requirement6 Requirement7 done
    • MorphBank: test Morphbank instance up and have procedures for building, installing, and populating with sample data. Issue 8 closed
    • Symbiota: test Symbiota instance up and have procedures for building, installing, and populating with sample data. Issue 10 closed
    • Specify: test Specify6 instance up and have procedures for building, installing, and populating with sample data. Issue 11 closed
  • Network
    • Access Point/Triage/Job Planner/Job Runner: Access point can accept an annotation message, job planner and job runner can produce plan and execute plan to publish annotation message. Network access point can generate GUIDs and implements sequence diagram for injection of messages. in memory stub only
    • Network Transport Layer = Overlay Network: Static configuration of capabilities needed for annotation plan. done, in memory Network interaction test suite with stubs for analysis, data store, and pub-sub capabilities. partial, needs analysis
    • Network Capabilities
      • Annotation Store: Redstore instance able to accept annotations. done
      • Harvesting and network data cache: Have sample data loaded by at least an ad-hoc means. not done'
      • Index: defined index specifications. not done
      • PubSub: stub, provides sample annotations, accepts annotation messages. done
      • Analysis: stub

The demonstration can be run as a multi-servlet web application, with one servlet for each data provider, for each index (if there is more than one), and one each for the pub-sub service and the client.

Still ToDo

  • Instantiation of network nodes.
  • Developer instructions for setting up network instance (instantiation network nodes).
  • Developer instructions for setting up network stub.
  • Revise UML model to reflect AO/AOD annotation.
  • Restrict web access on firuta.
  • Backing functionality for persistent user registration and Login/Logout on web client. Logout/Login in Use_Cases_from_Web_Client_Scenarios
  • Access point can accept an annotation message.
  • Job planner and job runner can produce plan and execute plan to publish annotation message. See: File:JobManagement.png In BOUML: Sequence diagram and Class diagram
  • Network access point can generate GUIDs and implements sequence diagram for injection of messages. File:HighLevelInteractionDiagram.png Also in BOUML Model. See also: File:3nodes.png, File:APIConcerns.png
  • Stub for analysis in network test suite.
  • Have sample data loaded into network cache/index by at least an ad-hoc means.

Next Steps

  • Specify Driver able to handle new determinations and update determinations. Issue 31
  • Refine WebUI for annotation processor with user input. Test workflow of login, viewing pending annotations, selecting annotation, processing new determination annotation. Likewise for update determination annotation. Issue 32 Issue 33
  • Generate response annotation from annotation processor. Issue 27 Issue 29
  • Create new determination annotation from Morphbank. Issue 30

2012-Mar-10

We have a Specify OAI-PMH service that can run in its own servlet and act as a data source wrapper for a Specify MySQL instance, using encrypted network transport Requirement14.

(Maybe: we have a client UI form for registering data sources)

(Maybe: our Symbiota client has a web form for annotating, analogous to our MorphBank's) Requirement22

Development goals:

  • Dev Environment: Ant tasks or other scripts for populating Specify, Symbiota, MorphBank, and index.
  • Documentation
    • UML Model for Network: Revisit Annotation and Message model.
    • FP Client API Definition
      • Annotation: AO/AOD reconciliation.
      • FP Messages: FP Messages for Ping, Annotation, and Subscription Requirement19 defined. Signature mechanism for FP Messages Defined. Requirement15 Message ID defined. Requirement16Soln
    • Code Generated javadocs.
    • Deployment instructions Add Developer instructions for setting up Specify-OAI and the index.
  • Clients
    • Annotation Processor: WebUI for annotation processor refined with user input from Jim and James. Able to use client helper to generate response annotation Requirement5Soln and wrap in FP annotation message. Requirement5Soln
    • Client Helper: Can generated fuzzed annotations for testing See: Scenarios for malicious uses. Able to create annotation documents for clients. Requirement7 Requirement6 Requirement11 Able to wrap annotation documents in FP Annotation messages Requirement16Soln. Able to generate FP_PING and FP_SUBSCRIBE Requirement19 messages
    • MorphBank: MorphBank client has a web form for eliciting a new determination from a user from a record displayed in its search results Requirement22 : using the Client helper it can compose the annotation rdf/xml and wrap it in a Filtered Push annotation message.
    • Web Client: Add web form for comparing two dwc_Occurrence objects side-by-side.
  • Network
    • Access Point/Triage/Job Planner/Job Runner: Basic implementation able to plan and run jobs for FP_PING, FP_ANNOTATION Requirement11, and FP_SUBSCRIBE Requirement19. Access point accepts signed Requirement15, encrypted Requirement14 messages with IDs Requirement16Soln.
    • Network Transport Layer = Overlay Network: Static configuration interacting on one machine.
    • Network Capabilities
      • Annotation Store: Able to put annotations into Redstore triple store. Requirement21
      • Harvesting and network data cache
        • Data Providers: Some work on Specify6 OAI-PMH service Requirement18 Requirement20
        • Harvester: Prototype OAI-PHM harvester that can be invoked to retrieve new and updated records from a data source over an encrypted connection Requirement14 and put them into the cache and index them Requirement18 Requirement20.
        • Network Cache & Index: Real implementation replaces the stub.

2012-Apr-10

MorphBank client can send new determination annotation messages containing a collection code in the subject to an access point and have them given to the pub-sub service have them retrieved by the annotation processor and ingested into specify as new determinations. Annotation processor able to generate and inject response annotations. Web client able to query annotation store for an annotation of a collection object and any response annotations connected to it.

  • Dev Environment: Able to instantiate clean network capabilities for an annotation store and a messaging service, populate them with test data, evaluate, and re-instantiate them as clean capabilities. Likewise morphbank and specify6.
    • Dependency Management: Maven or Ivy dependency management.
  • Documentation
    • UML Model for Network
    • FP Client API Definition Documented API for annotation generation using Client Helper
      • Annotation:
      • FP Messages: FP Message for query defined
    • Code Systematic javadoc comments and javadoc generation.
    • Deployment instructions Add developer instructions for setting up a network with a client, the cache+index, and JMS.
  • Clients
    • Annotation Processor:
      • Drivers:
    • Client Helper:
    • MorphBank: MorphBank web form for eliciting a new determination from a user refined from user input from Jim and James. Add ability to correct an existing determination in morphbank and sent an update determination annotation to network.
    • Symbiota:
    • Web Client: Able to query annotation store for annotations of a collection object and response annotations.
    • Specify:
    • Kepler Client:
  • Network
    • Access Point/Triage/Job Planner/Job Runner:
    • Network Transport Layer = Overlay Network: Static configuration of nodes able to communicate over the network, with the annotation storage capability at a different site from the pub-sub capability. Network transport between nodes is encrypted. Requirement14 Node configuration is static by system administrator(s).
    • Network Capabilities
      • Annotation Store: Redstore able to accept annotations and run sparql queries for validation.
      • Harvesting and network data cache
        • Data Providers:
        • Harvester:
        • Network Cache & Index:
      • PubSub: JMS-based pub-sub service can receive annotations and evaluate them to trigger appropriate subscriptions based on information present in annotations.
      • Analysis:
      • Generic Mapper:

2012-May-10

Symbiota and morphbank clients able to inject messages into multiple node network, have them delivered to relevant queues based on information in the messages and information in the network cache/index. Specify 6 able to ingest new determinations, updated determinations, and multiple other classes of annotations. Web client able to launch queries and retrieve results of combined information from both the network cache/index and the annotation store.

  • Dev Environment: Able to establish set of three nodes with annotation store, cache/index, and messaging capabilities distributed among nodes, along with Specify6 (for harvest), AnnotationProcessor with Specify6 driver, Morphbank, and WebClient clients to network.
    • Explore node deployment into iDigBio's VM/Appliance infrastructure.
    • Develop requirements for deploying an annotation generator as an appliance in iDigBio's infrastructure.
  • Documentation
    • UML Model for Network Rebuilt to reflect actual implementation.
    • FP Client API Definition Documented client API for Client
      • Annotation: Paper submitted. Example annotation document containing a georeference constructed with involvement of John W.
      • FP Messages: Defined message signatures and encryption for messages to and from network. Defined FP_SUBSCRIBE message.
    • Code
    • Deployment instructions Developer instructions for setting up a network with a client, the index, JMS, and the (a) mapper, and running the demonstration. Non-developer instructions for installing and running the demonstration.
  • Clients
    • Annotation Processor: The annotation processor can type and display annotations related to dwc_Occurrence, dwc_Identification, dwc_Taxon, and dwc_Location objects.
      • Drivers: The Specify driver can ingest core dwc_Occurrence, dwc_Location (locality and georeference) and dwc_Identification, dwc_Taxon (new/update determination) objects. See: Darwin_Core_to_Specify_Mapping
    • Client Helper: Add ability of client helper to generate query messages.
    • MorphBank: Web forms for transcribing core occurence and locality data from images of labels.
    • Symbiota: A web form for eliciting at least one form of annotation and invocation of the client helper able to inject this annotation into the network.
    • Web Client: Web client can receive notifications from the pub-sub service for logged-in users with subscriptions and display them in the subscription notification list UI and the notification display UI.
    • Specify:
    • Kepler Client:
  • Network
    • Access Point/Triage/Job Planner/Job Runner: Extend to include query job plans. Extend to include express interest job plans. Job plan for annotations extended to allow annotation delivery based on both annotation content and related data.
    • Network Transport Layer = Overlay Network: Implement capability discovery. Static configuration of list of participating nodes and clients.
    • Network Capabilities
      • Annotation Store: Deploy Annotation store more production ready than Redstore.
      • Harvesting and network data cache
        • Data Providers:
        • Harvester:
        • Network Cache & Index: Able to respond to queries in concert with the messaging system. Able to respond to query messages.
      • PubSub: Able to deliver query results to clients who made queries. Able to deliver annotations to interests based on information present in the annotation and on related information in the cache/index (in particular, able to deliver an annotation with an annotatesResource of just a GUID to the relevant collection database).
      • Analysis:
      • Generic Mapper:

2012-May-29

  • Code freeze for SPNHC Demo
  • Walk through the demonstration in the 2012May30 meeting.

2012-Jun-10

Add:

  • Documentation: Developer-level user-friendly documentation for downloading, building, installing, and running the demonstration.

Demonstration at SPNHC.

Components for SPNHC 2012 Demo, key components in blue or green.


The demonstration can be run temporarily on a minimum of three network nodes plus at least four clients as:

  1. a standalone MorphBank instance that can act as an annotation publication client Requirement22 Requirement23
  2. a standalone Symbiota instance that can act as an annotation publication client Requirement22 Requirement23
  3. a standalone web client instance that can issue queries, manage subscriptions Requirement19, and read notifications
  4. a network node providing a network access point, triage, job planning and job running
  5. an statically configured overlay network that allows network nodes to discover and access network capabilities
  6. network configured to encrypt all data in motion Requirement14
  7. a network capability providing a cache of data from multiple data providers with indexes
  8. a network capability providing a filtering pub-sub service Requirement20
  9. a network capability providing an annotation store
  10. implementation of job plans that allow
    1. publication of an annotation
    2. injection of query of the data cache and the annotation store (one time query)
    3. creation of an interest (subscription) expressed as a query (subscribe to a query) Requirement19
    4. delivery of annotations to interests based on both annotation content and related data Requirement20
    5. retrieval of accumulated messages on a subscription
  11. an annotation processing component that can be invoked to
    1. convert objects retrieved from a local data source to a standard network representation
    2. convert objects from a standard network representation to a representation ingestible local data source Requirement3
  12. an annotation processing user interface embedded in or configured to run alongside Specify6
    1. authorize and initiate the ingest of an object into a local data source Requirement3 Requirement4
    2. inform the network of actions taken in annotation processing Requirement5Soln

Subsequent Targets

2012-Jul-31

Able to demonstrate transcription of new specimen records from labels within morphbank, transport of these to Specify6, and ingestion of these new specimen records into Specify.

  • Dev Environment:
  • Documentation
  • Clients
    • Annotation Processor:
      • Drivers:
    • Client Helper:
    • MorphBank: Refine UI for new specimen data transcription with input from domain experts.
    • Symbiota:
    • Web Client:
    • Specify:
    • Kepler Client:
  • Network
    • Access Point/Triage/Job Planner/Job Runner: Extend job plans to executing analysies.
    • Network Transport Layer = Overlay Network:
    • Network Capabilities
      • Annotation Store:
      • Harvesting and network data cache
        • Data Providers:
        • Harvester:
        • Network Cache & Index:
      • PubSub:
      • Analysis: Kepler engine as network capability.
      • Generic Mapper:

2012-Aug-28

 

Early Fall 2012

  • Target Use Case Remote Data Capture
  • Transcribe new specimen record from label images in Morphbank.
  • Transport new specimen record annotation through FilteredPush network instance to relevant target collection database Requirement20.
  • View annotation in Mapper UI at instance of Specify6 database.
  • Accept and ingest new specimen record annotation into Specify 6 database, with resulting action annotation sent to network.
  • Implement automated ingest of authorized new specimen record annotations. Requirement31 Requirement32
  • Define a quality control test for harvested data, implement as a kepler workflow embedded in a network analytical capability.
  • Enable end user authentication through DataOne services. Requirement32 Requirement24
  • Network architecture uses configurable pipes rather than static node configuration.

2012-Dec-21

Blank Component List

  • Dev Environment:
  • Documentation
    • UML Model for Network
    • FP Client API Definition
      • Annotation:
      • FP Messages:
    • Code
    • Deployment instructions
  • Clients
    • Annotation Processor:
      • Drivers:
    • Client Helper:
    • MorphBank:
    • Symbiota:
    • Web Client:
    • Specify:
    • Kepler Client:
  • Network
    • Access Point/Triage/Job Planner/Job Runner:
    • Network Transport Layer = Overlay Network:
    • Network Capabilities
      • Annotation Store:
      • Harvesting and network data cache
        • Data Providers:
        • Harvester:
        • Network Cache & Index:
      • PubSub:
      • Analysis:
      • Generic Mapper:
Retrieved from "http://www.etaxonomy.org/mw/Roadmap"
Personal tools
All Hands Meeting