diff --git a/doc/README.md b/doc/README.md index 554ce397..6f131d9c 100644 --- a/doc/README.md +++ b/doc/README.md @@ -30,6 +30,14 @@ Historical documentation that is no longer relevant for current deployments: - **[INSTALL.md](archive/INSTALL.md)** - Original data loading environment setup (deprecated) - **[LOADING.md](archive/LOADING.md)** - TreeBASE v.1 to v.2 data migration procedures (deprecated) +### [v3/](v3/) + +UI/UX roadmap documentation for TreeBASE v3: + +- **[README.md](v3/README.md)** - Overview and links to all v3 user stories +- User stories covering search, account management, data submission, review, administration, + technical documentation, and governance + ## Quick Links ### I want to... @@ -41,6 +49,7 @@ Historical documentation that is no longer relevant for current deployments: - **Understand DWR integration** → [technical-notes/DWR.md](technical-notes/DWR.md) - **Learn about recent upgrades** → [technical-notes/UPGRADES.md](technical-notes/UPGRADES.md) - **Find historical data loading info** → [archive/](archive/) +- **See the v3 UI/UX roadmap** → [v3/](v3/) ## Contributing to Documentation diff --git a/doc/v3/README.md b/doc/v3/README.md new file mode 100644 index 00000000..b2cfc999 --- /dev/null +++ b/doc/v3/README.md @@ -0,0 +1,58 @@ +# TreeBASE v3 UI/UX Roadmap + +This directory contains the UI/UX roadmap documentation for TreeBASE v3. The goal is to define +clear user stories that map to current application pages, identify navigation flows, and +ensure all existing pages are accounted for. + +## Purpose + +The v3 roadmap aims to: +1. Document user stories for all major user types +2. Map each story to specific pages in the current application +3. Define navigation paths through the application +4. Prepare for future wireframes and mockups + +## User Stories + +### Public Users + +- [x] **[User Story 1: Search](user-story-01-search.md)** - A user or agent visits TreeBASE to search + for study, tree, matrix, taxon or analysis data + +### Authenticated Users + +- [x] **[User Story 2: Account Management](user-story-02-account.md)** - A user visits TreeBASE to + create or update their user account and log in + +- [x] **[User Story 3: Data Submission](user-story-03-submission.md)** - A user visits TreeBASE to + submit study, tree, matrix, taxon or analysis data + +### Reviewers + +- [x] **[User Story 4: Review](user-story-04-review.md)** - A reviewer visits TreeBASE to review a + study with tree, matrix, taxon and analysis data + +### Administrators + +- [x] **[User Story 5: Administration](user-story-05-admin.md)** - An admin visits TreeBASE to + administer the submission and review queue + +### Stakeholders + +- [x] **[User Story 6: Technical Documentation](user-story-06-technical.md)** - A technical + stakeholder visits TreeBASE to learn about the technology of the web app, database, data + standards, APIs, or other tools + +- [x] **[User Story 7: Governance](user-story-07-governance.md)** - A governance stakeholder visits + TreeBASE to learn about the project, its history, contributors, and role in phylogenetic data + management + +## Next Steps + +After these user stories are complete and all current pages are accounted for: + +1. Create wireframes, mockups, widgets and interactions for each user story +2. Design the updated house style and design principles +3. Plan the implementation phases +4. Phased implementation of user stories + diff --git a/doc/v3/user-story-01-search.md b/doc/v3/user-story-01-search.md new file mode 100644 index 00000000..537aa25a --- /dev/null +++ b/doc/v3/user-story-01-search.md @@ -0,0 +1,391 @@ +# User Story 01: Search + +## Overview + +**As a** user or automated agent, +**I want to** search for study, tree, matrix, taxon or analysis data, +**So that** I can find and access phylogenetic data relevant to my research or application. + +## User Types + +- Anonymous visitors (researchers, students, general public) +- Automated agents (harvesters, API clients, search engines) +- Registered users looking for data + +## Current Pages + +The following search pages are currently available in the TreeBASE web application: + +- [x] **Study Search Page** (`/search/studySearch.html`) - Search for phylogenetic studies +- [x] **Matrix Search Page** (`/search/matrixSearch.html`) - Search for character matrices +- [x] **Tree Search Page** (`/search/treeSearch.html`) - Search for phylogenetic trees +- [x] **Taxon Search Page** (`/search/taxonSearch.html`) - Search for taxonomic names +- [x] **Tree Topology Search Page** (`/search/treeTopSearch.html`) - Search by tree structure + +## Navigation Flow + +The following diagram shows how users navigate through search pages: + +```mermaid +graph TB + subgraph Entry Points + HOME[Home Page] + NAV[Navigation Bar] + DIRECT[Direct URL / API] + end + + subgraph Search Tabs + STUDY[Study Search
/search/studySearch.html] + MATRIX[Matrix Search
/search/matrixSearch.html] + TREE[Tree Search
/search/treeSearch.html] + TAXON[Taxon Search
/search/taxonSearch.html] + TREETOP[Tree Topology Search
/search/treeTopSearch.html] + end + + subgraph Search Forms + SIMPLE[Simple Search Form] + ADV[Advanced CQL Search] + end + + subgraph Results + RESULTS[Search Results List] + end + + subgraph Detail Pages + STUDY_DETAIL[Study Summary
/search/study/summary.html] + MATRIX_DETAIL[Matrix Detail
/search/study/matrix.html] + TREE_DETAIL[Tree Viewer
/search/study/tree.html] + TAXA_DETAIL[Taxa List
/search/study/taxa.html] + end + + subgraph Downloads + NEXUS[NEXUS Format] + NEXML[NeXML Format] + end + + HOME --> NAV + NAV --> STUDY + NAV --> MATRIX + NAV --> TREE + NAV --> TAXON + NAV --> TREETOP + DIRECT --> STUDY + DIRECT --> MATRIX + DIRECT --> TREE + DIRECT --> TAXON + DIRECT --> TREETOP + + STUDY --> SIMPLE + MATRIX --> SIMPLE + TREE --> SIMPLE + TAXON --> SIMPLE + TREETOP --> SIMPLE + + SIMPLE --> ADV + SIMPLE --> RESULTS + ADV --> RESULTS + + RESULTS --> STUDY_DETAIL + RESULTS --> MATRIX_DETAIL + RESULTS --> TREE_DETAIL + RESULTS --> TAXA_DETAIL + + STUDY_DETAIL --> NEXUS + STUDY_DETAIL --> NEXML + MATRIX_DETAIL --> NEXUS + TREE_DETAIL --> NEXUS + + %% Tab switching for result projection + RESULTS -.->|Project Results| STUDY + RESULTS -.->|Project Results| MATRIX + RESULTS -.->|Project Results| TREE + RESULTS -.->|Project Results| TAXON +``` + +### Navigation Notes + +- Users can switch between search tabs at any time via the navigation bar +- When results exist, switching tabs projects the current result set to the new data type +- Each search page provides both simple and advanced (CQL) search options +- Search results include links to detail pages and download options + +## Search Capabilities + +## Searchable Data Types + +TreeBASE supports searching across five primary data types. Each data type has specific searchable fields and display options. + +### 1. Studies + +Search for phylogenetic studies by metadata, authors, citations, etc. + +**Searchable Fields:** +| Field | Description | CQL Predicate | +|-------|-------------|---------------| +| Study ID | TreeBASE study identifier (S####) | `tb.identifier.study` | +| Legacy Study ID | TreeBASE 1.x study identifier | `tb.identifier.study.tb1` | +| Author | Author name(s) | `dcterms.contributor` | +| Title | Study/publication title | `tb.title.study` | +| Abstract | Publication abstract | `dcterms.abstract` | +| Citation | Full bibliographic citation | `dcterms.bibliographicCitation` | +| Keywords | Subject keywords | `dcterms.subject` | +| DOI | Digital Object Identifier | `prism.doi` | + +**Result Display Columns:** +- Study ID (with link to summary) +- Authors (first 3, then "et al.") +- Year +- Title +- Journal/Publisher +- DOI link (external) +- Download icons (NEXUS, NeXML) + +### 2. Matrices + +Search for character matrices used in phylogenetic analyses. + +**Searchable Fields:** +| Field | Description | CQL Predicate | +|-------|-------------|---------------| +| Matrix ID | TreeBASE matrix identifier (M####) | `tb.identifier.matrix` | +| Title | Matrix title/name | `tb.title.matrix` | +| Type | Matrix data type | `tb.type.matrix` | +| NTAX | Number of taxa | `tb.ntax.matrix` | +| NCHAR | Number of characters | `tb.nchar.matrix` | + +**Result Display Columns:** +- Matrix ID (with link to detail) +- Title +- Description +- Data Type +- NTAX +- NCHAR +- Download icons (original, reconstructed) +- View rows icon + +### 3. Trees + +Search for phylogenetic trees by properties, topology, or associated metadata. + +**Searchable Fields:** +| Field | Description | CQL Predicate | +|-------|-------------|---------------| +| Tree ID | TreeBASE tree identifier (Tr####) | `tb.identifier.tree` | +| Title | Tree title/name | `tb.title.tree` | +| Type | Tree type (e.g., Species Tree) | `tb.type.tree` | +| Kind | Tree kind (e.g., Consensus) | `tb.kind.tree` | +| Quality | Tree quality assessment | `tb.quality.tree` | +| NTAX | Number of taxa | `tb.ntax.tree` | + +**Result Display Columns:** +- Tree ID (with link to viewer) +- Label +- Title +- Tree Type +- Tree Kind +- Tree Quality +- NTAX +- View Taxa link +- Download icons (reconstructed, original) +- Tree viewer icon + +### 4. Taxa + +Search for taxonomic names and their occurrences across studies. + +**Searchable Fields:** +| Field | Description | CQL Predicate | +|-------|-------------|---------------| +| Taxon ID | TreeBASE taxon identifier (Tx####) | `tb.identifier.taxon` | +| Legacy Taxon ID | TreeBASE 1.x taxon identifier | `tb.identifier.taxon.tb1` | +| NCBI ID | NCBI Taxonomy identifier | `tb.identifier.ncbi` | +| uBio ID | uBio Namebank identifier | `tb.identifier.ubio` | +| Taxon Name | Scientific name | `tb.title.taxon` | +| Taxon Label | Label as used in study | `tb.title.taxonLabel` | +| Taxon Variant | Name variant | `tb.title.taxonVariant` | + +**Result Display Columns:** +- Taxon ID +- Taxon Name +- uBio ID (with external link) +- NCBI ID (with external link) + +### 5. Tree Topologies + +Search for trees based on their structural topology patterns. + +> **Note:** Tree Topology searches use a form-based interface rather than CQL queries. Users specify taxon names +> in a visual tree structure diagram, and the system finds trees matching that relationship pattern. + +**Search Types:** +| Search Type | Description | Input Fields | +|-------------|-------------|--------------| +| 3-Taxon Topology | Find trees containing a specific 3-taxon relationship | taxon_a, taxon_b, taxon_c | +| 4-Taxon Asymmetric | Find trees with asymmetric 4-taxon topology | taxon_a, taxon_b, taxon_c, taxon_d | +| 4-Taxon Symmetric | Find trees with symmetric 4-taxon topology | taxon_a, taxon_b, taxon_c, taxon_d | + +**Result Display:** Same as Trees (returns matching tree records) + +### 6. Classification Search (Secondary) + +The Classification Search (`/search/classificationSearch.html`) provides an alternative way to browse taxonomic +hierarchies. This is a specialized search interface that is not part of the main search tab navigation. + +**Note:** This feature is accessible via direct URL but is not prominently featured in the main search navigation. + +### Tab Navigation + +Each search type is accessible via a dedicated tab in the search interface. When a user selects a tab: + +- The search form updates to show relevant search fields for that data type +- An advanced search option is available for more complex queries using CQL +- The results page displays items of the selected type +- Pagination and sorting options are specific to the selected data type + +## Result Set Projection + +Users can project (transform) a result set from one data type into another. This allows users to explore related data +across different dimensions. + +### How Result Projection Works + +After performing a search and obtaining a result set, users can switch to a different data type tab to see the related +items. The system computes the union of all items of the new type that are associated with the original result set. + +### Example: Studies → Taxa + +1. **Initial Search**: User performs a study search and obtains a list of matching studies +2. **Projection**: User switches to the "Taxa" tab +3. **Result**: The system displays the union of all taxa that appear in any of the studies from the original result set + +### Supported Projections + +| From | To | Description | +|------|-----|-------------| +| Studies | Taxa | All taxa referenced in the selected studies | +| Studies | Matrices | All matrices associated with the selected studies | +| Studies | Trees | All trees generated by the selected studies | +| Studies | Tree Topologies | All tree topologies found in the selected studies | +| Matrices | Taxa | All taxa included in the selected matrices | +| Matrices | Studies | All studies that contain the selected matrices | +| Matrices | Trees | All trees derived from the selected matrices | +| Matrices | Tree Topologies | All tree topologies derived from the selected matrices | +| Trees | Taxa | All taxa present in the selected trees | +| Trees | Studies | All studies that produced the selected trees | +| Trees | Matrices | All matrices used to generate the selected trees | +| Trees | Tree Topologies | Topology patterns of the selected trees | +| Taxa | Studies | All studies that reference the selected taxa | +| Taxa | Matrices | All matrices containing the selected taxa | +| Taxa | Trees | All trees that include the selected taxa | +| Taxa | Tree Topologies | All tree topologies that include the selected taxa | +| Tree Topologies | Trees | All trees matching the selected topologies | +| Tree Topologies | Studies | All studies containing trees with the selected topologies | +| Tree Topologies | Matrices | All matrices that produced trees matching the selected topologies | +| Tree Topologies | Taxa | All taxa present in trees matching the selected topologies | + +### Use Cases + +- **Taxonomic Analysis**: Start with a study search, then project to taxa to understand which organisms are covered +- **Data Discovery**: Search for taxa, then project to studies to find relevant research +- **Tree Comparison**: Search for tree topologies, then project to studies to compare methodologies +- **Cross-Study Analysis**: Search for specific matrices, then project to taxa to see overlapping organisms + +## User Interface + +### Search Tabs + +``` +┌─────────┬──────────┬────────┬────────┬─────────────────┐ +│ Studies │ Matrices │ Trees │ Taxa │ Tree Topologies │ +└─────────┴──────────┴────────┴────────┴─────────────────┘ +``` + +Each tab is clickable and: +- Projects the current result set into the selected type when results from a previous search exist +- Performs a new search of that type when starting fresh (no previous results) + +A "Clear Results" option allows users to start a new search without projection. + +### Result Projection Indicator + +When viewing projected results, the interface should indicate: +- The original search type and query +- The current projection type +- The total number of items in both the original and projected result sets + +## Technical Notes + +- Result projection is computed dynamically based on database relationships +- Large result sets (over 1,000 items) may trigger a warning to users about potential delay when projecting +- Pagination applies to displayed results; projection computes against the complete result set +- Cache strategies may be employed to improve projection performance for frequently accessed result sets +- Maximum result set size for projection is configurable by administrators + +## Pages to Account For + +Complete inventory of pages related to search functionality: + +### Main Search Pages + +| Page | URL Pattern | Description | Status | +|------|-------------|-------------|--------| +| Study Search | `/search/studySearch.html` | Main study search interface | Active | +| Matrix Search | `/search/matrixSearch.html` | Matrix search interface | Active | +| Tree Search | `/search/treeSearch.html` | Tree search interface | Active | +| Taxon Search | `/search/taxonSearch.html` | Taxon search interface | Active | +| Tree Topology Search | `/search/treeTopSearch.html` | Topology-based tree search | Active | + +### Secondary Search Pages + +| Page | URL Pattern | Description | Status | +|------|-------------|-------------|--------| +| Classification Search | `/search/classificationSearch.html` | Hierarchical classification browse (not in main nav) | Active | + +### Study Detail Pages + +| Page | URL Pattern | Description | Status | +|------|-------------|-------------|--------| +| Study Summary | `/search/study/summary.html` | Study detail/summary view | Active | +| Study Matrices | `/search/study/matrices.html` | Matrices within a study | Active | +| Study Trees | `/search/study/trees.html` | Trees within a study | Active | +| Study Taxa | `/search/study/taxa.html` | Taxa within a study | Active | +| Study Analyses | `/search/study/analyses.html` | Analyses within a study | Active | +| Study Analysis Detail | `/search/study/analysis.html` | Single analysis detail | Active | +| Matrix Detail | `/search/study/matrix.html` | Matrix detail view | Active | +| Tree Viewer | `/search/study/tree.html` | Interactive tree viewer | Active | +| Tree Block Viewer | `/search/study/treeBlock.html` | Tree block viewer | Active | +| Tree Blocks List | `/search/study/treeBlocks.html` | List of tree blocks | Active | +| Row Segments | `/search/study/rowSegments.html` | Matrix row segments | Active | + +### Download Endpoints + +| Page | URL Pattern | Description | Status | +|------|-------------|-------------|--------| +| Row Segments TSV | `/search/study/rowSegmentsTSV.html` | Row segments as TSV download | Active | +| RDF Export | `/search/study/anyObjectAsRDF.rdf` | RDF export for any object | Active | +| Download Study | `/search/downloadAStudy.html` | Download complete study | Active | +| Download Tree | `/search/downloadATree.html` | Download single tree | Active | +| Download Tree Block | `/search/downloadATreeBlock.html` | Download tree block | Active | +| Download Matrix | `/search/downloadAMatrix.html` | Download single matrix | Active | +| Download Analysis Step | `/search/downloadAnAnalysisStep.html` | Download analysis step data | Active | +| Download NEXUS | `/search/downloadANexusFile.html` | Download original NEXUS file | Active | +| Download NEXUS RCT | `/search/downloadANexusRCTFile.html` | Download reconstructed NEXUS file | Active | + +### PhyloWS API Endpoints + +| Endpoint | Description | +|----------|-------------| +| `/phylows/study/find` | Study search API | +| `/phylows/matrix/find` | Matrix search API | +| `/phylows/tree/find` | Tree search API | +| `/phylows/taxon/find` | Taxon search API | + +## Wireframe Notes + +*To be completed in future PR* + +## Open Questions + +- What search refinement options should be available? +- How should results be sorted and paginated? +- What metadata should be shown in search results? \ No newline at end of file diff --git a/doc/v3/user-story-02-account.md b/doc/v3/user-story-02-account.md new file mode 100644 index 00000000..e1d755b9 --- /dev/null +++ b/doc/v3/user-story-02-account.md @@ -0,0 +1,225 @@ +# User Story 2: Account Management + +## Overview + +**As a** user, +**I want to** create or update my user account and log in, +**So that** I can access authenticated features like data submission and study management. + +## User Types + +- New users creating an account +- Existing users logging in +- Users updating their profile information +- Users managing their password/credentials + +## Current Pages + +The following pages apply to this user story: + +- [x] Registration page (`/register.html` → `userForm.jsp`) +- [x] Login page (`/login.jsp` → `loginForm.jsp`) +- [x] Logout functionality (`/logout` → Spring Security logout handler) +- [x] User profile page (`/user/updateProfile.html` → `userForm.jsp`) +- [x] Password reset request page (`/passwordForm.html` → `passwordForm.jsp`) +- [x] Password reset form page (`/resetPassword.html` → `resetPasswordForm.jsp`) +- [x] Admin user management pages (`/admin/overrideUserProfile.html`, `/admin/adminSelectUsers.html`, etc.) + +## Navigation Flow + +The following diagram documents how users navigate through account management pages: + +```mermaid +flowchart TD + subgraph Public["Public Pages"] + Home["/home.html
Home Page"] + Login["/login.jsp
Login Page
(loginForm.jsp)"] + Register["/register.html
Registration Page
(userForm.jsp)"] + PwdRequest["/passwordForm.html
Password Reset Request
(passwordForm.jsp)"] + PwdReset["/resetPassword.html
Reset Password Form
(resetPasswordForm.jsp)"] + end + + subgraph Authenticated["Authenticated Pages"] + UserHome["/user/processUser.html
User Landing"] + Profile["/user/updateProfile.html
Update Profile
(userForm.jsp)"] + Submissions["/user/submissionList.html
My Submissions"] + Logout["/logout
Logout Handler"] + end + + subgraph Admin["Admin Pages (Admin/Associate Editor only)"] + AdminPage["/admin/administrationPage.html
Admin Home"] + SelectUsers["/admin/adminSelectUsers.html
Select Users"] + OverrideProfile["/admin/overrideUserProfile.html
Override User Profile"] + MergeUsers["/admin/adminMergingUsers.html
Merge Users"] + DeleteUser["/admin/adminDeletingUserStepOne.html
Delete User"] + end + + Home -->|"Login link"| Login + Login -->|"Sign up link"| Register + Login -->|"Forgot password"| PwdRequest + Login -->|"Submit credentials
(j_security_check)"| UserHome + Register -->|"Submit registration"| Login + PwdRequest -->|"Request sent"| PwdRequest + PwdRequest -->|"Email link"| PwdReset + PwdReset -->|"Password reset success"| Login + + UserHome -->|"Navigation tab"| Profile + UserHome -->|"Navigation tab"| Submissions + Profile -->|"Update success"| Profile + Submissions -->|"Logout link"| Logout + Profile -->|"Logout link"| Logout + Logout --> Login + + UserHome -->|"Admin tab
(if Admin role)"| AdminPage + AdminPage --> SelectUsers + SelectUsers --> OverrideProfile + AdminPage --> MergeUsers + AdminPage --> DeleteUser +``` + +## Account Features + +### Registration + +Registration is handled by `RegisterUserController` with the following requirements from the current implementation: + +**Required Fields:** +- **Username** - Unique identifier for login (required, unique in database, `user.username`) +- **Password** - User's password (required, BCrypt encoded before storage) +- **Re-typed Password** - Password confirmation (required, must match password) +- **First Name** - User's first name (required, `user.firstName`) +- **Last Name** - User's last name (required, `user.lastName`) +- **Email Address** - Contact email (required, unique in database, `user.emailAddressString`) + +**Optional Fields:** +- **Middle Name** - User's middle name (`user.middleName`) +- **Phone Number** - Contact phone number (`user.phoneNumber`) + +**Not Implemented:** +- Institutional affiliation - Not present in current User entity + +**Registration Flow:** +1. User navigates to `/register.html` +2. User fills in required and optional fields +3. Client-side JavaScript validates password match via `checkPasswords()` +4. Server-side validation via `BeanValidator` checks field constraints +5. Password is encoded using BCrypt via `PasswordEncoder` +6. `UserService.createUser()` checks for duplicate username/email +7. On success, redirect to `/login.jsp` +8. On failure (duplicate username/email), error message displayed + +### Authentication + +Authentication is implemented using Spring Security 5.8.15 (`treebase-security.xml`): + +**Login Mechanism:** +- Form-based authentication at `/j_security_check` +- Username parameter: `j_username` +- Password parameter: `j_password` +- Login page: `/login.jsp` +- Default success URL: `/user/processUser.html` +- Failure URL: `/login.jsp?error=true` + +**Password Handling:** +- `DelegatingPasswordEncoder` for backward compatibility (supports BCrypt and legacy plain text) +- `PasswordUpgradeAuthenticationSuccessHandler` automatically upgrades plain text passwords to BCrypt on successful login + +**Session Management:** +- Session fixation protection: `newSession` strategy +- Session timeout: Managed by servlet container (Tomcat) defaults +- CSRF protection: Currently disabled for backward compatibility + +**Authorization (URL-based):** +- `/submit.html` - Requires: User, Admin, or Associate Editor +- `/user/**` - Requires: User, Admin, or Associate Editor +- `/admin/**` - Requires: Admin or Associate Editor + +**Role Configuration:** +- Role prefix: Empty string (no `ROLE_` prefix) +- Existing roles in database: `Admin`, `User`, `Associate Editor` + +**Not Implemented:** +- Remember me functionality - Not present in current security configuration +- Logout from all devices - Single session logout only via `/logout` + +### Profile Management + +Profile management is handled by `UserFormController` with the following functionality: + +**View/Update Profile:** +- URL: `/user/updateProfile.html` +- Controller: `UserFormController` +- View: `userForm.jsp` + +**Editable Fields:** +- **Password** - Can update password (leave blank to keep current) +- **Re-typed Password** - Must match new password if changing +- **First Name** - User's first name +- **Middle Name** - User's middle name (optional) +- **Last Name** - User's last name +- **Phone Number** - Contact phone number (optional) +- **Email Address** - Contact email + +**Read-only Fields:** +- **Username** - Cannot be changed after registration + +**Security:** +- Users can only modify their own profile (enforced in `UserFormController`) +- Admin users can modify other users' profiles via `/admin/overrideUserProfile.html` +- Password changes are validated and BCrypt-encoded before storage + +**View Submission History:** +- URL: `/user/submissionList.html` +- Shows list of user's submissions with their status +- Each submission links to detailed view and editing options + +**Not Implemented:** +- Change email address - Email is editable but not re-verified +- Manage notification preferences - Not present in current implementation + +## Pages to Account For + +Complete inventory of pages related to account management from `/treebase-web/src/main/webapp/WEB-INF/treebase-servlet.xml`: + +| Page | URL Pattern | Controller | View | Status | +|------|-------------|------------|------|--------| +| Registration | `/register.html` | `registerUserController` | `userForm.jsp` | Active | +| Login | `/login.jsp` | (static JSP) | `login.jsp` | Active | +| Login Form | (included in login.jsp) | - | `loginForm.jsp` | Active | +| Password Reset Request | `/passwordForm.html` | `passwordFormController` | `passwordForm.jsp` | Active | +| Password Reset Form | `/resetPassword.html` | `resetPasswordController` | `resetPasswordForm.jsp` | Active | +| User Profile | `/user/updateProfile.html` | `userFormController` | `userForm.jsp` | Active | +| Process User (Post-login) | `/user/processUser.html` | `processUserController` | - | Active | +| Submission List | `/user/submissionList.html` | `listSubmissionController` | - | Active | +| Logout | `/logout` | Spring Security | Redirect to `/login.jsp` | Active | +| Admin - Select Users | `/admin/adminSelectUsers.html` | `adminSelectUsersController` | `adminSelectUsers.jsp` | Active | +| Admin - User List | `/admin/userList.html` | `filenameController` | `userList.jsp` | Active | +| Admin - Override Profile | `/admin/overrideUserProfile.html` | `adminOverridingUserFormController` | `overrideUserProfile.jsp` | Active | +| Admin - Update User Info | `/admin/adminUpdatingUserInfo.html` | `adminUpdatingUserInfoController` | `adminUpdatingUserInfo.jsp` | Active | +| Admin - Delete User Step 1 | `/admin/adminDeletingUserStepOne.html` | `adminDeletingUserStepOneController` | `adminDeletingUserStepOne.jsp` | Active | +| Admin - Delete User Step 2 | `/admin/adminDeletingUserStepTwo.html` | `adminDeletingUserStepTwoController` | `adminDeletingUserStepTwo.jsp` | Active | +| Admin - Merge Users | `/admin/adminMergingUsers.html` | `adminMergingUsersController` | `adminMergingUsers.jsp` | Active | +| Admin - User Management | `/admin/userManagement.html` | `userManagementController` | `simpleUserManagement.jsp` | Active | + +## Security Considerations + +Current implementation status: + +- **Password strength requirements**: Minimum 8 characters enforced in password reset form; registration form uses JavaScript validation but could be enhanced with server-side enforcement +- **Account lockout policies**: Not currently implemented; consider adding after N failed attempts +- **Session timeout**: Managed by servlet container (Tomcat) defaults +- **HTTPS requirements**: Should be enforced at server/proxy level; not enforced in application code +- **Email verification**: Not currently implemented; email is validated but not verified via confirmation link +- **Password reset token security**: + - Tokens are cryptographically secure (UUID-based) + - 24-hour expiration + - Single-use (marked as used after successful reset) + - Previous tokens invalidated when new one is generated + - Timing-attack protection via normalized response times +- **CSRF protection**: Currently disabled; should be enabled in future security hardening +- **Role-based access control**: Implemented via Spring Security with URL-based authorization + +## Wireframe Notes + +*To be completed in future PR* + diff --git a/doc/v3/user-story-03-submission.md b/doc/v3/user-story-03-submission.md new file mode 100644 index 00000000..632af4f5 --- /dev/null +++ b/doc/v3/user-story-03-submission.md @@ -0,0 +1,611 @@ +# User Story 3: Data Submission + +## Overview + +**As a** registered user, +**I want to** submit study, tree, matrix, taxon or analysis data, +**So that** I can share my phylogenetic research with the scientific community. + +## User Types + +- Researchers submitting their own studies +- Students submitting work on behalf of advisors +- Data curators submitting historical data + +## Prerequisites + +- User must be logged in (see [User Story 2: Account Management](user-story-02-account.md)) + +## Current Pages + +The following pages apply to this user story: + +- [x] Submission dashboard (`/user/submissionList.html` → `submissionList.jsp`) +- [x] New submission form (`/user/studyForm.html?form=new` → `studyForm.jsp`) +- [x] Study summary/metadata (`/user/summary.html` → `submissionSummaryView.jsp`) +- [x] Citation form (`/user/citationForm.html` → `citationForm.jsp`) +- [x] Author search/add (`/user/authorSearchForm.html`, `/user/addAuthor.html` → `peopleSearchForm.jsp`, `addPersonForm.jsp`) +- [x] File upload interface (`/user/uploadFile.html` → `uploadFile.jsp`) +- [x] Upload summary (`/user/uploadFileSummary.html` → `uploadFileSummary.jsp`) +- [x] Tree blocks list (`/user/treeBlockList.html` → `treeBlockList.jsp`) +- [x] Tree list/editor (`/user/treeList.html` → `treeList.jsp`) +- [x] Tree viewer/editor (`/user/directMapToPhyloWidget.html` → `treeViewer.jsp`) +- [x] Matrix list/editor (`/user/matrixList.html` → `matrixList.jsp`) +- [x] Matrix row list (`/user/matrixRowList.html` → `matrixRowList.jsp`) +- [x] Row segment list for matrix row (`/user/matrixRowSegmentList.html` → `matrixRowSegmentList.jsp`) +- [x] Row segment form (`/user/matrixRowSegmentForm.html` → `matrixRowSegmentForm.jsp`) +- [x] Upload row segment data (`/user/uploadRowSegmentData.html` → `uploadRowSegmentData.jsp`) +- [x] Row segment data table (`/user/rowSegmentDataTable.html` → `rowSegmentDataTable.jsp`) +- [x] View all row segment data (`/user/viewAllRowSegmentData.html` → `viewAllRowSegmentData.jsp`) +- [x] Export row segment data (`/user/exportRowSegmentData.html` → file download) +- [x] Export row segment template (`/user/exportRowSegmentTemplate.html` → file download) +- [x] Delete row segment (`/user/deleteARowSegment.html` → `generalDeletePage.jsp`) +- [x] Taxa management (`/user/taxaList.html` → `taxonLabels.jsp`) +- [x] Edit taxon label (`/user/editTaxonLabel.html` → `editTaxonLabel.jsp`) +- [x] Analysis display (`/user/analysisDisplay.html` → `analysisList.jsp`) +- [x] Analysis form (`/user/analysisForm.html` → `analysisForm.jsp`) +- [x] Analysis step form (`/user/analysisStepForm.html` → `analysisStepForm.jsp`) +- [x] Analyzed data form (`/user/analyzedDataForm.html` → `analyzedDataForm.jsp`) +- [x] Ready state confirmation (`/user/readyState.html` → `readyState.jsp`) +- [x] Delete study (`/user/deleteStudy.html` → `deleteStudy.jsp`) +- [x] Submission tutorial (`/submitTutorial.html` → `submitTutorial.jsp`) + +## Navigation Flow + +The following diagram documents how users navigate through the submission pages: + +```mermaid +flowchart TD + subgraph Dashboard["Submission Dashboard"] + SubmitList["/user/submissionList.html
My Submissions
(submissionList.jsp)"] + end + + subgraph Create["Create Submission"] + NewStudy["/user/studyForm.html?form=new
New Submission Form
(studyForm.jsp)"] + end + + subgraph StudyMeta["Study Metadata"] + Summary["/user/summary.html
Submission Summary
(submissionSummaryView.jsp)"] + Citation["/user/citationForm.html
Citation Form
(citationForm.jsp)"] + AuthorSearch["/user/authorSearchForm.html
Author Search
(peopleSearchForm.jsp)"] + AddAuthor["/user/addAuthor.html
Add Author
(addPersonForm.jsp)"] + end + + subgraph Upload["File Upload"] + UploadFile["/user/uploadFile.html
NEXUS File Upload
(uploadFile.jsp)"] + UploadSummary["/user/uploadFileSummary.html
Upload Summary
(uploadFileSummary.jsp)"] + end + + subgraph Trees["Tree Management"] + TreeBlocks["/user/treeBlockList.html
Tree Blocks
(treeBlockList.jsp)"] + TreeList["/user/treeList.html
Trees in Block
(treeList.jsp)"] + TreeViewer["/user/directMapToPhyloWidget.html
Tree Viewer
(treeViewer.jsp)"] + end + + subgraph Matrices["Matrix Management"] + MatrixList["/user/matrixList.html
Matrices
(matrixList.jsp)"] + MatrixRows["/user/matrixRowList.html
Matrix Rows
(matrixRowList.jsp)"] + end + + subgraph RowSegments["Row Segment Management"] + RowSegmentList["/user/matrixRowSegmentList.html
Row Segments for Row
(matrixRowSegmentList.jsp)"] + RowSegmentForm["/user/matrixRowSegmentForm.html
Edit Row Segment
(matrixRowSegmentForm.jsp)"] + UploadRowSegment["/user/uploadRowSegmentData.html
Upload Row Segment Data
(uploadRowSegmentData.jsp)"] + RowSegmentDataTable["/user/rowSegmentDataTable.html
Map Columns
(rowSegmentDataTable.jsp)"] + ViewAllRowSegments["/user/viewAllRowSegmentData.html
All Row Segments
(viewAllRowSegmentData.jsp)"] + ExportRowSegment["/user/exportRowSegmentData.html
Export TSV
(file download)"] + ExportTemplate["/user/exportRowSegmentTemplate.html
Export Template
(file download)"] + DeleteRowSegment["/user/deleteARowSegment.html
Delete Row Segment
(generalDeletePage.jsp)"] + end + + subgraph Taxa["Taxon Management"] + TaxaList["/user/taxaList.html
Taxon Labels
(taxonLabels.jsp)"] + EditTaxon["/user/editTaxonLabel.html
Edit Taxon
(editTaxonLabel.jsp)"] + end + + subgraph Analysis["Analysis Management"] + AnalysisList["/user/analysisDisplay.html
Analyses
(analysisList.jsp)"] + AnalysisForm["/user/analysisForm.html
Analysis Form
(analysisForm.jsp)"] + AnalysisStep["/user/analysisStepForm.html
Analysis Step
(analysisStepForm.jsp)"] + AnalyzedData["/user/analyzedDataForm.html
Analyzed Data
(analyzedDataForm.jsp)"] + end + + subgraph Submit["Review & Submit"] + ReadyState["/user/readyState.html
Change to Ready State
(readyState.jsp)"] + DeleteStudy["/user/deleteStudy.html
Delete Study
(deleteStudy.jsp)"] + end + + %% Main flow + SubmitList -->|"New Submission"| NewStudy + NewStudy -->|"Submit"| Summary + SubmitList -->|"Edit submission"| Summary + + %% Summary navigation (via Tool Box menu) + Summary -->|"Citation"| Citation + Summary -->|"Upload"| UploadFile + Summary -->|"Trees"| TreeBlocks + Summary -->|"Matrices"| MatrixList + Summary -->|"Taxa"| TaxaList + Summary -->|"Analysis"| AnalysisList + + %% Citation flow + Citation -->|"Add/search authors"| AuthorSearch + AuthorSearch -->|"Add new"| AddAuthor + AddAuthor --> AuthorSearch + Citation -->|"Done"| Summary + + %% Upload flow + UploadFile -->|"Upload complete"| UploadSummary + UploadSummary -->|"Continue"| Summary + + %% Tree flow + TreeBlocks -->|"View trees"| TreeList + TreeList -->|"Edit tree"| TreeViewer + TreeViewer --> TreeList + + %% Matrix flow + MatrixList -->|"View rows"| MatrixRows + MatrixRows -->|"View segments"| RowSegmentList + + %% Row Segment flow (individual row) + RowSegmentList -->|"Add segment"| RowSegmentForm + RowSegmentList -->|"Edit segment"| RowSegmentForm + RowSegmentForm -->|"Done"| RowSegmentList + + %% Row Segment bulk operations flow + MatrixList -->|"Upload row segment data"| UploadRowSegment + UploadRowSegment -->|"Upload complete"| RowSegmentDataTable + RowSegmentDataTable -->|"Submit"| ViewAllRowSegments + ViewAllRowSegments -->|"Export"| ExportRowSegment + ViewAllRowSegments -->|"Upload more"| UploadRowSegment + ViewAllRowSegments -->|"Delete segment"| DeleteRowSegment + ViewAllRowSegments -->|"Back to rows"| MatrixRows + MatrixList -->|"Export template"| ExportTemplate + DeleteRowSegment -->|"Confirmed"| ViewAllRowSegments + + %% Taxa flow + TaxaList -->|"Edit taxon"| EditTaxon + EditTaxon --> TaxaList + TaxaList -->|"Validate labels"| TaxaList + + %% Analysis flow + AnalysisList -->|"Add analysis"| AnalysisForm + AnalysisForm -->|"Add step"| AnalysisStep + AnalysisStep -->|"Add data"| AnalyzedData + AnalyzedData --> AnalysisStep + AnalysisStep --> AnalysisForm + AnalysisForm --> AnalysisList + + %% Submission status + SubmitList -->|"Change status"| ReadyState + ReadyState -->|"Success"| SubmitList + SubmitList -->|"Delete"| DeleteStudy + DeleteStudy -->|"Confirmed"| SubmitList +``` + +## Submission Workflow + +### Step 1: Study Information + +Study information is recorded on multiple pages managed by `StudyFormController` and `CitationFormController`: + +**Study Form** (`/user/studyForm.html` → `studyForm.jsp`): +- **Study Name** (required) - Brief title, usually same as publication title +- **Study Notes** - Private notes for submitter/staff communication, not public + +**Citation Form** (`/user/citationForm.html` → `citationForm.jsp`): +- **Citation Type** (required) - Article, Book, or Book Section +- **Publish Year** (required) - Year of publication (1900 to current+1) +- **Publication Status** - Status of the publication + +For Article citations (`citationForm-article.jsp`): +- **Article Title** (required) +- **Keywords** - Comma-separated keywords +- **Abstract** - Publication abstract +- **PubMed ID** (PMID) - Maximum 10 characters +- **DOI** - Digital Object Identifier +- **URL** - Link to publication +- **Journal** (required) - Autocomplete from existing journals +- **Volume** +- **Issue** +- **Pages** - In format like "11-28" + +**Author Management** (`/user/authorSearchForm.html`, `/user/addAuthor.html`): +- Search for existing authors or add new ones +- Authors linked to citation via `AuthorFormController` + +### Step 2: Data Upload + +File upload is handled by `UploadFileController` (`/user/uploadFile.html` → `uploadFile.jsp`): + +**Supported Formats:** +- **NEXUS** - Primary format, parsed by Mesquite +- Files must be parseable by Mesquite before upload + +**Upload Interface:** +- Multiple file upload support via "Attach another file" +- Progress bar during upload +- **Maximum upload size**: 250MB (configured in `treebase-servlet.xml`) + +**Upload Constraints:** +- Only first ~30 trees are parsed to avoid overwhelming the interface +- Users should place preferred/consensus trees first in the tree block + +**Upload Summary** (`/user/uploadFileSummary.html` → `uploadFileSummary.jsp`): +- Shows count of matrices uploaded +- Shows count of trees uploaded +- Parser log available via "Show Parser Log" button + +### Step 3: Tree Data + +Tree management is handled by `ListTreeBlockController` and `ListTreeController`: + +**Tree Block List** (`/user/treeBlockList.html` → `treeBlockList.jsp`): +- **Block Title** - Editable title for the tree block +- **Tree Count** - Number of trees in the block +- **Analysis Status** - Icon showing if block is linked to an analysis +- Actions: View trees, View in PhyloWidget, Download, Delete + +**Tree List** (`/user/treeList.html` → `treeList.jsp`): +- **Tree Label** - Editable label +- **Tree Title** - Editable title +- **Tree Type** - Dropdown selection (e.g., Consensus, Single) +- **Tree Kind** - Dropdown selection (e.g., Species Tree, Gene Tree) +- **Tree Quality** - Dropdown selection +- **Analysis Status** - Icon showing if tree is linked to an analysis +- Actions: Edit in PhyloWidget, Download reconstructed, Download original NEXUS, Delete + +**Tree Viewer** (`/user/directMapToPhyloWidget.html` → `treeViewer.jsp`): +- PhyloWidget-based tree visualization and editing +- Opens in popup window (1000x900 pixels) + +### Step 4: Matrix Data + +Matrix management is handled by `ListMatrixController`: + +**Matrix List** (`/user/matrixList.html` → `matrixList.jsp`): +- **Matrix Title** - Editable title +- **Matrix Description** - Editable description +- **Matrix Kind** - Dropdown selection (e.g., DNA, Protein, Morphological) +- **Analysis Status** - Icon showing if matrix is linked to an analysis +- Actions: View rows, Download reconstructed, Download original NEXUS, Delete + +**Matrix Row List** (`/user/matrixRowList.html` → `matrixRowList.jsp`): +- View matrix rows +- Navigate to row segments for each row + +### Step 4a: Row Segment Data + +Row segments represent parts of sequences within character state matrices. They allow users to annotate specific portions of matrix rows with specimen information, accession numbers, and geographic data. + +**Example Use Case:** A researcher has a concatenated DNA matrix where positions 1-500 are 16S rRNA and positions 501-1200 are COI. Using row segments, they can annotate these regions separately for each taxon, recording different GenBank accession numbers and specimen voucher codes for each gene fragment. + +**Row segment management provides two approaches:** + +#### Individual Row Segment Entry + +Via `ListMatrixRowSegmentController` and `SingleRowSegmentController`: + +**Row Segment List** (`/user/matrixRowSegmentList.html` → `matrixRowSegmentList.jsp`): +- Lists all row segments for a specific matrix row +- Shows segment title, start index, and end index +- Actions: Add new segment, Edit segment + +**Row Segment Form** (`/user/matrixRowSegmentForm.html` → `matrixRowSegmentForm.jsp`): +- **Taxon Label** - Display only, inherited from matrix row +- **Title** - Segment title/name +- **Start Index** / **End Index** - Character positions defining the segment +- **Segment Data** - Selected portion of the sequence (auto-filled from selection) + +Specimen Label fields: +- **Institution Acronym** - Museum/institution code +- **Collection Code** - Collection identifier +- **Catalog Number** - Specimen catalog number +- **GenBank Accession** - GenBank accession number +- **Other Accession** - Other database accession numbers +- **Sample Date** - Collection date (format: YYYY-MM-DD) +- **Collector** - Name of collector + +Geographic fields: +- **Latitude** / **Longitude** - Decimal coordinates +- **Elevation** - Elevation in meters +- **Country** / **State** / **Locality** - Administrative location + +- **Notes** - Free-text notes about the specimen + +Interactive segment selection: Users can highlight text in the sequence display and click "Select" to auto-populate start/end indices. + +#### Bulk Row Segment Upload + +Via `UploadRowSegmentDataController`, `RowSegmentDataTableController`, and `ViewAllRowSegmentDataController`: + +**Upload Row Segment Data** (`/user/uploadRowSegmentData.html` → `uploadRowSegmentData.jsp`): +- Upload tab-delimited files containing row segment data for multiple rows +- File should contain columns matching row segment fields + +**Row Segment Data Table** (`/user/rowSegmentDataTable.html` → `rowSegmentDataTable.jsp`): +- Preview of uploaded data (first 10 rows displayed) +- Column mapping interface to match file columns to row segment fields +- Available field mappings: + - Taxon Label + - Segment Title + - Start Index / End Index + - Institution Acronym, Collection Code, Catalog Number + - GenBank Accession, Other Accession + - Sample Date, Collector + - Latitude, Longitude, Elevation + - Country, State, Locality + - Notes + - Ignore (skip column) +- Checkbox to indicate if file includes header row +- Validates for duplicate column selections + +**View All Row Segments** (`/user/viewAllRowSegmentData.html` → `viewAllRowSegmentData.jsp`): +- Displays all row segments for the current matrix +- Inline editing of all segment fields +- Bulk update: Modify multiple segments and save changes +- Bulk delete: Select segments via checkboxes and delete +- Links to: + - Export Row Segment Data + - Upload Another Row Segment Data File + - Go Back To Matrix Rows + +**Export Row Segment Data** (`/user/exportRowSegmentData.html`): +- Downloads current row segment data as tab-separated values (TSV) file +- Filename: `RowSegmentData_{matrixId}.tsv` +- Handled by `ExportRowSegmentDataController` + +**Export Row Segment Template** (`/user/exportRowSegmentTemplate.html`): +- Downloads template file with headers for bulk upload +- Filename: `RowSegmentTemplate_{matrixId}.txt` +- Handled by `ExportRowSegmentTemplateController` + +**Delete Row Segment** (`/user/deleteARowSegment.html` → `generalDeletePage.jsp`): +- Confirmation page before deleting individual row segment +- Requires WRITE permission on the submission +- Handled by `DeleteARowSegmentController` + +### Step 5: Taxonomic Data + +Taxon management is handled by `ListTaxaController` and `EditTaxonLabelController`: + +**Taxon Labels List** (`/user/taxaList.html` → `taxonLabels.jsp`): +- **Taxon Label** - Original label from data file +- **Taxon Name** - Mapped scientific name +- **NCBI Tax ID** - Link to NCBI Taxonomy Browser +- **uBio Tax ID** - Link to uBio NameBank +- **Validation Status** - Icon showing if linking was attempted +- Actions: Edit, Bulk update, Validate selected labels + +**Edit Taxon Label** (`/user/editTaxonLabel.html` → `editTaxonLabel.jsp`): +- **Taxon Label** - Editable text, guidelines provided for formatting +- **External Taxonomy** - Radio selection from suggested matches: + - NCBI taxonomy suggestions with IDs + - uBio lookup option with manual NamebankID entry + - "No association" option if no match exists + +**Validation Process:** +- Labels validated against uBio and NCBI taxonomies +- Guidelines: Use full binomials (no abbreviations), separate codes with spaces +- "Validate Taxon Labels" button triggers bulk validation + +### Step 6: Analysis Information + +Analysis management is handled by `AnalysisFormController` and `AnalysisStepFormController`: + +**Analysis Form** (`/user/analysisForm.html` → `analysisForm.jsp`): +- **Analysis Name** - Descriptive name for the analysis +- **Analysis Notes** - Additional notes about the analysis + +**Analysis Step Form** (`/user/analysisStepForm.html` → `analysisStepForm.jsp`): +- **Step Name** - Name for this analysis step +- **Step Notes** - Additional notes +- **Commands** - Actual commands/settings used +- **Software Name** - Autocomplete from known software +- **Software Version** - Version number +- **Software URL** - Link to software +- **Software Description** - Brief description +- **Algorithm Type** - Dropdown: Maximum Likelihood, Parsimony, Distance, Bayesian, Other +- **New Algorithm** - Text field when "Other Algorithm" selected + +**Analyzed Data** (`/user/analyzedDataForm.html` → `analyzedDataForm.jsp`): +- Multi-page wizard for linking inputs/outputs: + 1. Select data type and input/output designation + 2. Select matrices (for input) + 3. Select trees (for output) + 4. Select tree blocks + +### Step 7: Review and Submit + +The review and submission process validates data completeness before allowing status change: + +**Submission Summary** (`/user/summary.html` → `submissionSummaryView.jsp`): +- Displays submission number and status +- Shows study accession URL (permanent identifier) +- Shows reviewer access URL with access code +- Citation summary if entered +- Abstract if provided +- Analysis list embedded + +**Ready State Confirmation** (`/user/readyState.html` → `readyState.jsp`): + +**Pre-submission Validation Checks:** +The following must be satisfied before changing to "Ready State": +1. Citation entered with minimum: authors, year, title, journal name +2. No orphaned matrices (all matrices must be inputs to at least one analysis) +3. No orphaned trees (all trees must be outputs from at least one analysis) +4. Tree taxon labels must match input matrix taxon labels +5. Taxon labels validation attempted (via "Validate Taxon Labels" button) + +**Submission Confirmation:** +- Warning message about read-only status after submission +- Exception: Citation metadata can still be updated (volume, issue, pages, DOI) +- Submit button completes the status change + +## Data Validation + +Data validation occurs at multiple stages in the submission process: + +### File Format Validation + +**At Upload Time:** +- NEXUS files are parsed using Mesquite library +- Invalid syntax results in parse errors shown in Parser Log +- Recommendation: Test files in Mesquite before upload + +**File Size Validation:** +- Maximum upload size: 250MB (configurable in `treebase-servlet.xml`) +- Tree block limit: Only first ~30 trees parsed per block + +### Required Field Validation + +**Study Form:** +- Study Name: Required, validated server-side + +**Citation Form:** +- Citation Type: Required (Article, Book, Book Section) +- Publish Year: Required, validated 1900 ≤ year ≤ current+1 +- Title: Required for articles +- Journal: Required for articles +- At least one author required (validated at Ready State) + +**Analysis Step Form:** +- Algorithm Type validation when "Other" selected - New Algorithm field required + +**Row Segment Form:** +- Start Index: Must be 1 or greater and must not exceed the total sequence length +- End Index: Must be greater than or equal to Start Index and must not exceed the total sequence length +- Segment coordinates validated against matrix row data + +### Taxonomic Name Validation + +**Automated Validation:** +- Triggered by "Validate Taxon Labels" button +- Matches labels against uBio and NCBI taxonomies +- Visual indicators show validation status: + - Green checkmark: Linking attempted + - Red X: Not yet validated + +**Manual Validation:** +- Edit individual labels via Edit Taxon Label page +- Search uBio manually and enter NamebankID +- Select "no association" for unmatched taxa + +### Consistency Checks (at Ready State) + +The system validates the following before allowing status change: + +1. **Citation Completeness:** + - Authors present + - Year present + - Title present + - Journal name present (for articles) + +2. **Data Completeness:** + - No orphaned matrices (each matrix linked to ≥1 analysis as input) + - No orphaned trees (each tree linked to ≥1 analysis as output) + +3. **Taxon Label Matching:** + - Output tree taxon labels must exist in input matrix + - Mismatched labels prevent Ready State + +4. **Validation Attempted:** + - Taxon label validation must be attempted + +**Error Display:** +- Errors shown in red at top of Ready State page +- Tool Box items highlighted in yellow indicate problems +- Hover over highlighted items for error details + +## Pages to Account For + +Complete inventory of pages related to data submission from `/treebase-web/src/main/webapp/WEB-INF/treebase-servlet.xml`: + +| Page | URL Pattern | Controller | View | Status | +|------|-------------|------------|------|--------| +| Submission List | `/user/submissionList.html` | `listSubmissionController` | `submissionList.jsp` | Active | +| New Study Form | `/user/studyForm.html` | `studyFormController` | `studyForm.jsp` | Active | +| Submission Summary | `/user/summary.html` | `summaryController` | `submissionSummaryView.jsp` | Active | +| Citation Form | `/user/citationForm.html` | `citationFormController` | `citationForm.jsp` | Active | +| Author Search | `/user/authorSearchForm.html` | `authorSearchFormController` | `peopleSearchForm.jsp` | Active | +| Add Author | `/user/addAuthor.html` | `addAuthorController` | `addPersonForm.jsp` | Active | +| Upload File | `/user/uploadFile.html` | `uploadFileController` | `uploadFile.jsp` | Active | +| Upload Summary | `/user/uploadFileSummary.html` | `uploadFileSummaryController` | `uploadFileSummary.jsp` | Active | +| Tree Block List | `/user/treeBlockList.html` | `listTreeBlockController` | `treeBlockList.jsp` | Active | +| Tree List | `/user/treeList.html` | `listTreeController` | `treeList.jsp` | Active | +| Tree Viewer | `/user/directMapToPhyloWidget.html` | `directMapToPhyloWidgetController` | `treeViewer.jsp` | Active | +| Delete Tree | `/user/deleteATree.html` | `deleteATreeController` | `generalDeletePage.jsp` | Active | +| Delete Tree Block | `/user/deleteATreeBlock.html` | `deleteATreeBlockController` | `generalDeletePage.jsp` | Active | +| Matrix List | `/user/matrixList.html` | `listMatrixController` | `matrixList.jsp` | Active | +| Matrix Row List | `/user/matrixRowList.html` | `listMatrixRowController` | `matrixRowList.jsp` | Active | +| Row Segment List | `/user/matrixRowSegmentList.html` | `listMatrixRowSegmentController` | `matrixRowSegmentList.jsp` | Active | +| Row Segment Form | `/user/matrixRowSegmentForm.html` | `matrixRowSegmentFormController` | `matrixRowSegmentForm.jsp` | Active | +| Upload Row Segment Data | `/user/uploadRowSegmentData.html` | `uploadRowSegmentDataController` | `uploadRowSegmentData.jsp` | Active | +| Row Segment Data Table | `/user/rowSegmentDataTable.html` | `rowSegmentDataTableController` | `rowSegmentDataTable.jsp` | Active | +| View All Row Segments | `/user/viewAllRowSegmentData.html` | `viewAllRowSegmentDataController` | `viewAllRowSegmentData.jsp` | Active | +| Export Row Segment Data | `/user/exportRowSegmentData.html` | `exportRowSegmentDataController` | (file download) | Active | +| Export Row Segment Template | `/user/exportRowSegmentTemplate.html` | `exportRowSegmentTemplateController` | (file download) | Active | +| Delete Row Segment | `/user/deleteARowSegment.html` | `deleteARowSegmentController` | `generalDeletePage.jsp` | Active | +| Delete Matrix | `/user/deleteAMatrix.html` | `deleteAMatrixController` | `generalDeletePage.jsp` | Active | +| Taxa List | `/user/taxaList.html` | `listTaxaController` | `taxonLabels.jsp` | Active | +| Edit Taxon Label | `/user/editTaxonLabel.html` | `editTaxonLabelController` | `editTaxonLabel.jsp` | Active | +| Edit Set Taxon Labels | `/user/editSetTaxonLabel.html` | `editSetTaxonLabelController` | `editSetTaxonLabel.jsp` | Active | +| Analysis List | `/user/analysisList.html` | `listAnalysisController` | `analysisList.jsp` | Active | +| Analysis Display | `/user/analysisDisplay.html` | `displayAnalysisController` | `analysisList.jsp` | Active | +| Analysis Form | `/user/analysisForm.html` | `analysisFormController` | `analysisForm.jsp` | Active | +| Analysis Step List | `/user/analysisStepList.html` | `listAnalysisStepController` | `analysisStepList.jsp` | Active | +| Analysis Step Form | `/user/analysisStepForm.html` | `analysisStepFormController` | `analysisStepForm.jsp` | Active | +| Analyzed Data List | `/user/analyzedDataList.html` | `listAnalyzedDataController` | `analyzedDataList.jsp` | Active | +| Analyzed Data Form | `/user/analyzedDataForm.html` | `analyzedDataFormController` | `analyzedDataForm.jsp` | Active | +| Add Analyzed Data | `/user/addAnalyzedData.html` | `addAnalyzedDataController` | - | Active | +| Ready State | `/user/readyState.html` | `readyStateController` | `readyState.jsp` | Active | +| Delete Study | `/user/deleteStudy.html` | `deleteStudyController` | `deleteStudy.jsp` | Active | +| Submit Tutorial | `/submitTutorial.html` | `filenameController` | `submitTutorial.jsp` | Active | +| Download Tree | `/user/downloadATree.html` | `downloadATreeController` | - | Active | +| Download Tree Block | `/user/downloadATreeBlock.html` | `downloadATreeBlockController` | - | Active | +| Download Matrix | `/user/downloadAMatrix.html` | `downloadAMatrixController` | - | Active | +| Download NEXUS File | `/user/downloadANexusFile.html` | `downloadANexusFileController` | - | Active | + +## Wireframe Notes + +*To be completed in future PR* + +## Open Questions + +The following questions have been addressed based on current implementation: + +- **What file formats should be supported?** + - NEXUS is the primary supported format + - Files must be parseable by Mesquite + - TreeBASE does not directly support Newick-only files; they should be embedded in NEXUS format + +- **What is the maximum file size?** + - 250MB maximum upload size (configured in `treebase-servlet.xml`) + - Only first ~30 trees per block are parsed + +- **Can submissions be saved as drafts?** + - Yes, submissions in "In Progress" status function as drafts + - Users can continue editing until they change status to "Ready" + - "In Progress" submissions are not visible to the public + +- **What validation happens at each step vs. at submission?** + - File validation: At upload time (NEXUS parsing) + - Field validation: On form submission (server-side Spring validators) + - Comprehensive validation: When changing to "Ready State" (orphan checks, taxon matching, etc.) + +- **How are row segments used?** + - Row segments annotate portions of character state matrix sequences + - They link specimen information (museum codes, GenBank accessions) to specific sequence regions + - Can be entered individually per row or uploaded in bulk via tab-delimited files + - Templates can be exported to facilitate bulk data entry + +## Security Considerations + +- **File Upload Security**: File uploads handled by `AjaxMultipartResolver` with size limits +- **Access Control**: Submission pages require User, Admin, or Associate Editor role +- **Study Ownership**: Users can only modify their own submissions (enforced in controllers) +- **Status Transitions**: Only study owners and admins can change submission status +- **Read-only After Ready**: Once in "Ready" status, only citation metadata can be updated +- **Delete Confirmation**: Study deletion requires explicit confirmation via `deleteStudy.jsp` confirmation page with warning message about permanent data loss +- **Delete Restrictions**: Studies can only be deleted when in "In Progress" status; all trees and matrices must be deleted first before the study can be removed +- **Row Segment Authorization**: Row segment deletion requires WRITE permission on the containing submission. WRITE permission is granted to the study owner and administrators. Users can verify their permission by checking if edit/delete buttons appear on the row segment pages. diff --git a/doc/v3/user-story-04-review.md b/doc/v3/user-story-04-review.md new file mode 100644 index 00000000..ea20c2a5 --- /dev/null +++ b/doc/v3/user-story-04-review.md @@ -0,0 +1,178 @@ +# User Story 4: Review + +## Overview + +**As a** reviewer, +**I want to** review a study with tree, matrix, taxon and analysis data, +**So that** I can verify the quality and accuracy of submitted phylogenetic data. + +## User Types + +- Journal editors reviewing submissions +- Peer reviewers assigned to studies +- TreeBASE curators performing quality control + +## Prerequisites + +- Data submitter has provided the reviewer with a URL with a special access + token in the query string (`x-access-code` parameter) +- Reviewer visits the website via this URL to access the unpublished study + +## Current Pages + +The following pages are involved in the review workflow: + +- [x] Reviewer access agreement popup (`nav.jsp` lines 191-257) - Modal dialog that appears on first access requiring agreement to confidentiality terms +- [x] Study summary page (`search/study/summary.html`) - Displays citation, authors, abstract, keywords, and external links +- [x] Trees list page (`search/study/trees.html`) - Lists all trees in the study with download options +- [x] Tree viewer (`treeViewer.jsp`) - Interactive phylogenetic tree visualization using phylotree.js +- [x] Matrices list page (`search/study/matrices.html`) - Lists all character matrices with metadata +- [x] Matrix detail page (`search/study/matrix.html`) - Shows matrix row data +- [x] Taxa list page (`search/study/taxa.html`) - Shows taxon labels with NCBI and uBio taxonomy links +- [x] Analyses list page (`search/study/analyses.html`) - Shows analysis steps, software, and data connections + +## Navigation Flow + +```mermaid +flowchart TD + A[Reviewer receives URL
with x-access-code] --> B{First Visit?} + B -->|Yes| C[Reviewer Access
Agreement Popup] + C -->|Accept| D[Study Summary Page] + C -->|Cancel| Z[Access Denied] + B -->|No| D + + D --> E[Navigation Menu] + E --> F[Citation/Summary] + E --> G[Taxa] + E --> H[Matrices] + E --> I[Trees] + E --> J[Analyses] + + F --> K[View Authors] + F --> L[View Abstract] + F --> M[Download Options
Nexus/NeXML] + + G --> N[Taxon Labels List] + N --> O[NCBI Taxonomy Links] + N --> P[uBio Links] + + H --> Q[Matrix List] + Q --> R[Matrix Detail View] + Q --> S[Download Matrix
Nexus/NeXML] + Q --> T[View Taxa for Matrix] + + I --> U[Tree List] + U --> V[Tree Viewer
phylotree.js] + U --> W[Download Tree
Nexus/NeXML] + U --> X[View Taxa for Tree] + + J --> Y[Analysis Steps] + Y --> AA[Software Info] + Y --> AB[Input/Output Data Links] +``` + +## Review Workflow + +The reviewer navigates through study data using the x-access-code URL parameter which grants temporary read-only access to unpublished submissions. + +### Step 1: Initial Access & Agreement +1. Reviewer receives URL with `x-access-code` parameter from journal editor or submitter +2. URL format: `/treebase-web/phylows/study/TB2:S{id}?x-access-code={token}&format=html` +3. On first access, a modal agreement dialog appears requiring acceptance of: + - Confidentiality of data + - Agreement not to retain data after review + - Agreement not to use data for own research until published + - Agreement to keep URL confidential +4. Clicking "OK" stores session acceptance and allows access +5. A red banner "You are in reviewer mode" appears on all pages + +### Step 2: Study Overview Review +- View study summary page with citation information +- Check publication details (authors, year, journal, title) +- Verify metadata completeness (DOI, PMID, abstract, keywords) +- Review author contact information +- Access BibTeX and RIS reference formats + +### Step 3: Tree Data Review +- Navigate to Trees tab to see all trees in the study +- For each tree, verify: + - Tree label and title + - Tree type (e.g., species tree, gene tree) + - Tree kind (e.g., consensus, bootstrap) + - Number of taxa +- Use interactive tree viewer (phylotree.js) to: + - Visualize tree topology + - Hover over nodes for branch length information + - Verify taxon labels match expected names +- Download trees in Nexus or NeXML format for local verification + +### Step 4: Matrix Data Review +- Navigate to Matrices tab +- For each matrix, review: + - Matrix title and description + - Data type (DNA, RNA, protein, morphological, etc.) + - Number of taxa (NTAX) and characters (NCHAR) +- Access matrix row view to check: + - Character data completeness + - Taxon label consistency +- Download matrices for local software verification + +### Step 5: Taxonomic Review +- Navigate to Taxa tab (accessible from study level or per-tree/matrix) +- Verify taxon labels are properly mapped +- Check NCBI Taxonomy ID links for validation +- Check uBio NameBank ID links +- Flag any nomenclatural issues or unmapped labels + +### Step 6: Analysis Review +- Navigate to Analyses tab +- For each analysis, verify: + - Software used (name and version) + - Algorithm type (parsimony, likelihood, Bayesian, etc.) + - Command strings (PAUP blocks, MrBayes commands) + - Input data (matrices) linked correctly + - Output data (trees) linked correctly +- Ensure analysis steps are complete and reproducible + +## Pages to Account For + +| Page | URL Pattern | Status | +| ---- | ----------- | ------ | +| Study Summary | `/search/study/summary.html?id={studyId}` | ✅ Implemented | +| Trees List | `/search/study/trees.html?id={studyId}` | ✅ Implemented | +| Tree Viewer | `/search/study/tree.html?treeid={treeId}` | ✅ Implemented | +| Matrices List | `/search/study/matrices.html?id={studyId}` | ✅ Implemented | +| Matrix Detail | `/search/study/matrix.html?id={studyId}&matrixid={matrixId}` | ✅ Implemented | +| Taxa List | `/search/study/taxa.html?id={studyId}` | ✅ Implemented | +| Taxa for Tree | `/search/study/taxa.html?id={studyId}&treeid={treeId}` | ✅ Implemented | +| Taxa for Matrix | `/search/study/taxa.html?id={studyId}&matrixid={matrixId}` | ✅ Implemented | +| Analyses List | `/search/study/analyses.html?id={studyId}` | ✅ Implemented | +| Analysis Detail | `/search/study/analysis.html?id={studyId}&analysisid={analysisId}` | ✅ Implemented | +| Download Study (Nexus) | `/phylows/study/TB2:S{id}?format=nexus&x-access-code={token}` | ✅ Implemented | +| Download Study (NeXML) | `/phylows/study/TB2:S{id}?format=nexml&x-access-code={token}` | ✅ Implemented | + +## Wireframe Notes + +*To be completed in future PR* + +## Open Questions + +- What review statuses are needed? +- How should reviewer assignments work? +- What communication happens between reviewer and submitter? +- Can reviews be collaborative (multiple reviewers)? + +## Technical Implementation Notes + +### Access Control +- Access tokens are MD5 hashes generated from the study's namespaced GUID (e.g., "TB2:S1234") +- The token is obtained via `study.namespacedGUID.getHashedIDString()` method in Java, or `${submission.study.namespacedGUID.hashedIDString}` in JSP +- Token is passed via `x-access-code` URL parameter +- Token is stored in session after first successful access +- Constant defined in `Constants.java`: `X_ACCESS_CODE = "x-access-code"` + +### Key Files +- `nav.jsp` - Navigation and reviewer agreement popup +- `PhyloWSController.java` - Handles access code validation and URL generation +- `submissionSummaryView.jsp` - Shows reviewer access URL to submitters +- `Constants.java` - Defines access code constant diff --git a/doc/v3/user-story-05-admin.md b/doc/v3/user-story-05-admin.md new file mode 100644 index 00000000..bfc8c17f --- /dev/null +++ b/doc/v3/user-story-05-admin.md @@ -0,0 +1,243 @@ +# User Story 5: Administration + +## Overview + +**As an** administrator, +**I want to** administer studies and users, +**So that** I can ensure efficient processing of phylogenetic data submissions. + +## User Types + +- System administrators +- TreeBASE curators with admin privileges (Admin role) + +## Prerequisites + +- User must be logged in with Admin role +- Access is controlled via Spring Security configuration in `treebase-security.xml` + +## Current Pages + +The following pages are involved in the administration workflow: + +- [x] Admin landing page (`administrationPage.jsp`) - Main dashboard with links to Study Management and User Management menus +- [x] Study management for a user (`simpleUserManagement.jsp`) - List studies by submitter with filters +- [x] Management of ready studies (`changeStudyStatus.jsp`) - View and change status of Ready state studies +- [x] Search submissions (`searchBySubmissionID.jsp`) - Search by TreeBASE2, TB1 legacy, or TB2 Study ID +- [x] Select studies (`selectStudies.jsp`) - Select studies by status type +- [x] Select users (`adminSelectUsers.jsp`) - Search users by email, username, last name, or role +- [x] Update user info (`adminUpdatingUserInfo.jsp`) - Enter username to update +- [x] Override user profile (`overrideUserProfile.jsp`) - Modify user details after lookup +- [x] Delete user Step 1 (`adminDeletingUserStepOne.jsp`) - Enter username to delete +- [x] Delete user Step 2 (`adminDeletingUserStepTwo.jsp`) - Confirm deletion +- [x] Merge users (`adminMergingUsers.jsp`) - Merge source user into target user +- [x] Select persons (`adminSelectPersons.jsp`) - Find potential duplicate person records +- [x] Merge persons (`adminMergingPersons.jsp`) - Merge person records (affects user accounts and citations) +- [x] Merge persons Step 2 (`adminMergingPersonsStepTwo.jsp`) - Confirm person merge + +## Navigation Flow + +```mermaid +flowchart TD + A[Login with Admin/Associate Editor Role] --> B[Administration Page] + + subgraph StudyMgmt["Study Management Menu"] + B --> C[For a User] + B --> D[Ready State Studies] + B --> E[Search by Submission ID] + B --> F[Select Studies] + end + + subgraph UserMgmt["User Management Menu"] + B --> G[Select Users] + B --> H[Update User Info] + B --> I[Delete User] + B --> J[Merge Users] + B --> K[Select Persons] + B --> L[Merge Persons] + end + + C --> M[Change Study Status Page] + D --> M + E --> M + F --> M + + M --> N{Action} + N -->|Change Status| O[Update Status Radio] + N -->|Delete| P[Delete Study] + N -->|View| Q[Study Summary] + + G --> R[User List] + H --> S[Override User Profile] + I --> T[Delete User Step 1] + T --> U[Delete User Step 2] + U --> V[Message After Action] + + J --> W[Merge Users Wizard] + W --> V + + K --> X[Person List - Duplicates] + L --> Y[Merge Persons Step 1] + Y --> Z[Merge Persons Step 2] + Z --> V +``` + +## Administration Functions + +### Study Management + +#### For a User (`/admin/userManagement.html`) +List all studies submitted by a particular user. + +| Field | Type | Options/Values | +|-------|------|----------------| +| Search by | Radio | Email Address, Username (default), Last Name | +| User info | Text | Search term matching selected criteria | +| Study type | Dropdown | All, In Progress, Ready (default), Published | + +#### Ready State Studies (`/admin/readyStateStudies.html`) +Shows all studies currently in "Ready" state for curation review. + +#### Search by Submission ID (`/admin/searchBySubmissionID.html`) +Search for submissions using various identifier types. + +| Field | Type | Options/Values | +|-------|------|----------------| +| Identifier Type | Radio | TreeBASE2 Submission ID (default), TreeBASE1 Legacy Study ID, TreeBase2 Study ID | +| Submission Accession | Text | The ID value to search (max 25 chars) | + +#### Select Studies (`/admin/selectStudies.html`) +Filter studies by their status. + +| Field | Type | Options/Values | +|-------|------|----------------| +| Study type | Dropdown | All, In Progress, Ready (default), Published | + +#### Change Study Status (`/admin/changeStudyStatus.html`) +View and modify study statuses. Displayed as a table with columns: +- ID (link to study summary) +- Submitter (email link) +- Study Name +- Study Notes +- Created Date +- Last Modified Date +- Change Status (radio buttons: In Progress, Ready, Published) +- Delete link + +### User Management + +#### Select Users (`/admin/adminSelectUsers.html`) +Search for users based on various criteria. + +| Field | Type | Options/Values | +|-------|------|----------------| +| Search by | Radio | Email Address, Username (default), Last Name, User Role | +| User info | Text | Search term (for Email/Username/Last Name) | +| User Role | Dropdown | (Available roles from system, e.g., Admin, User, Associate Editor) | + +#### Update User Info (`/admin/adminUpdatingUserInfo.html`) +Enter a username to look up and modify. + +| Field | Type | Description | +|-------|------|-------------| +| Username | Text | Username of the account to update | + +Redirects to Override User Profile page on success. + +#### Delete User (`/admin/adminDeletingUserStepOne.html` → `adminDeletingUserStepTwo.html`) +Two-step process to delete a user account. + +**Step 1:** +| Field | Type | Description | +|-------|------|-------------| +| Username | Text | Username of the account to delete | + +**Step 2:** Confirmation page before deletion. + +#### Merge Users (`/admin/adminMergingUsers.html`) +Combine two user accounts into one. + +| Field | Type | Description | +|-------|------|-------------| +| Source Username | Text | User account to be merged (will be removed) | +| Target Username | Text | User account to receive merged data | + +#### Select Persons (`/admin/adminSelectPersons.html`) +Audit tool to find potential duplicate person records. + +| Field | Type | Options/Values | +|-------|------|----------------| +| Search criteria | Radio | With Same First and Last Name (default), With Same Last Name | + +#### Merge Persons (`/admin/adminMergingPersons.html` → `adminMergingPersonsStepTwo.html`) +Merge duplicate person records. This affects: +- User accounts associated with the person records +- Citation author/editor records + +**Step 1:** +| Field | Type | Description | +|-------|------|-------------| +| Source Person ID | Text | ID of person record to be merged (will be deleted) | +| Target Person ID | Text | ID of person record to receive merged data | + +**Step 2:** Confirmation page showing what will be affected. + +## Pages to Account For + +| Page | URL Pattern | JSP File | Status | +|------|-------------|----------|--------| +| Administration Landing | `/admin/administrationPage.html` | `administrationPage.jsp` | ✅ Implemented | +| User Management (Studies) | `/admin/userManagement.html` | `simpleUserManagement.jsp` | ✅ Implemented | +| Ready State Studies | `/admin/readyStateStudies.html` | `changeStudyStatus.jsp` | ✅ Implemented | +| Search by Submission ID | `/admin/searchBySubmissionID.html` | `searchBySubmissionID.jsp` | ✅ Implemented | +| Select Studies | `/admin/selectStudies.html` | `selectStudies.jsp` | ✅ Implemented | +| Change Study Status | `/admin/changeStudyStatus.html` | `changeStudyStatus.jsp` | ✅ Implemented | +| Select Users | `/admin/adminSelectUsers.html` | `adminSelectUsers.jsp` | ✅ Implemented | +| User List | `/admin/userList.html` | `userList.jsp` | ✅ Implemented | +| Update User Info | `/admin/adminUpdatingUserInfo.html` | `adminUpdatingUserInfo.jsp` | ✅ Implemented | +| Override User Profile | `/admin/overrideUserProfile.html` | `overrideUserProfile.jsp` | ✅ Implemented | +| Delete User Step 1 | `/admin/adminDeletingUserStepOne.html` | `adminDeletingUserStepOne.jsp` | ✅ Implemented | +| Delete User Step 2 | `/admin/adminDeletingUserStepTwo.html` | `adminDeletingUserStepTwo.jsp` | ✅ Implemented | +| Merge Users | `/admin/adminMergingUsers.html` | `adminMergingUsers.jsp` | ✅ Implemented | +| Select Persons | `/admin/adminSelectPersons.html` | `adminSelectPersons.jsp` | ✅ Implemented | +| Person List | `/admin/personList.html` | `personList.jsp` | ✅ Implemented | +| Merge Persons | `/admin/adminMergingPersons.html` | `adminMergingPersons.jsp` | ✅ Implemented | +| Merge Persons Step 2 | `/admin/adminMergingPersons.html` | `adminMergingPersonsStepTwo.jsp` | ✅ Implemented | +| Message After Action | `/admin/messageToAdminAfterAction.html` | `messageToAdminAfterAction.jsp` | ✅ Implemented | + +## Technical Implementation Notes + +### Access Control +- Admin pages are protected by Spring Security (`treebase-security.xml`) +- URL pattern `/admin/**` requires `Admin` or `Associate Editor` role +- Menu visibility controlled via `menu-config.xml` with `roles="Admin,Associate Editor"` + +### Key Configuration Files +- `menu-config.xml` - Defines Study Management and User Management menus +- `treebase-servlet.xml` - URL to controller mappings (lines 1020-1037) +- `treebase-security.xml` - Security intercept rules + +### Controllers +- `userManagementController` - For a User page +- `selectStudiesController` - Select Studies page +- `changeStudyStatusController` - Ready State and Change Status pages +- `searchBySubmissionIDController` - Search by Submission ID +- `adminSelectUsersController` - Select Users +- `adminUpdatingUserInfoController` - Update User Info +- `adminOverridingUserFormController` - Override User Profile +- `adminDeletingUserStepOneController` / `adminDeletingUserStepTwoController` - Delete User +- `adminMergingUsersController` - Merge Users +- `adminSelectPersonsController` - Select Persons +- `adminMergingPersonsController` - Merge Persons + +## Wireframe Notes + +*To be completed in future PR* + +## Open Questions + +- What admin roles/permission levels are needed beyond Admin and Associate Editor? +- What metrics should be tracked on the dashboard? +- How should escalation workflows work for stuck submissions? +- What audit logging requirements exist for admin actions? +- Should there be bulk operations for study status changes? diff --git a/doc/v3/user-story-06-technical.md b/doc/v3/user-story-06-technical.md new file mode 100644 index 00000000..3fc9d01c --- /dev/null +++ b/doc/v3/user-story-06-technical.md @@ -0,0 +1,294 @@ +# User Story 6: Technical Documentation + +## Overview + +**As a** technical stakeholder, +**I want to** learn about the technology of the web app, database, data standards, APIs, or other +tools, +**So that** I can integrate with TreeBASE, contribute to development, or understand its technical +architecture. + +## User Types + +- Software developers integrating with TreeBASE +- API consumers building applications +- Contributors to TreeBASE development +- Researchers using programmatic access +- System administrators deploying TreeBASE + +## Current Pages + +The following JSP pages and documentation files serve technical stakeholders: + +### Web Application Pages (JSP) +- [x] **urlAPI.jsp** (`/urlAPI.html`) - Data Access page with PhyloWS API overview, URI patterns, and RSS feeds +- [x] **technology.jsp** (`/technology.html`) - Implementation technologies and architecture overview +- [x] **about.jsp** (`/about.html`) - Background, history, funding, and related resources +- [x] **contact.jsp** (`/contact.html`) - Helpdesk contact and GitHub issue tracker links +- [x] **submitTutorial.jsp** (`/submitTutorial.html`) - NEXUS file preparation and submission process +- [x] **help.jsp** (`/help.html`) - Dynamic help text system + +### Documentation Files (Markdown) +- [x] **doc/API.md** - Comprehensive PhyloWS API documentation with CQL search syntax +- [x] **doc/OAI-PMH.md** - OAI-PMH metadata harvesting service documentation +- [x] **doc/development/BUILDING.md** - Build prerequisites and instructions (Java 17, Maven) +- [x] **doc/development/DEPLOYING.md** - Tomcat deployment guide with troubleshooting +- [x] **doc/technical-notes/DWR.md** - DWR (Direct Web Remoting) integration for AJAX +- [x] **doc/technical-notes/UPGRADES.md** - Dependency upgrade history (Spring, Jersey, etc.) + +### External Resources +- [ ] GitHub Wiki - API documentation (linked from urlAPI.jsp) - *External, not maintained in repository* +- [ ] GitHub Wiki - OAI-PMH documentation (linked from urlAPI.jsp) - *External, not maintained in repository* + +## Navigation Flow + +Technical users navigate through documentation pages based on their integration needs: + +```mermaid +flowchart TD + A[Entry Points] --> B{User Goal} + + B -->|API Integration| C["urlAPI.html (Data Access)"] + B -->|Understand System| D["technology.html (Implementation)"] + B -->|Contribute Code| E[GitHub Repository] + B -->|Deploy Instance| F["doc/development/"] + + C --> G["doc/API.md (PhyloWS Reference)"] + C --> H["doc/OAI-PMH.md (Harvesting Interface)"] + + G --> I["phylows/study endpoint"] + G --> J["phylows/tree endpoint"] + G --> K["phylows/matrix endpoint"] + G --> L["phylows/taxon endpoint"] + + D --> M["about.html (Background)"] + D --> N[Architecture Diagram] + + E --> O["doc/development/BUILDING.md"] + E --> P["doc/technical-notes/"] + + F --> O + F --> Q["doc/development/DEPLOYING.md"] + + H --> R[OAI-PMH Verbs] + + style C fill:#e1f5fe + style D fill:#e8f5e9 + style E fill:#fff3e0 + style F fill:#fce4ec +``` + +## Technical Documentation Areas + +### API Documentation + +Available documentation for API consumers: + +| Resource | Location | Description | +|----------|----------|-------------| +| **urlAPI.jsp** | `/urlAPI.html` | Overview of programmatic access, URI patterns, RSS feeds | +| **doc/API.md** | Repository | PhyloWS API reference with CQL search syntax and examples | +| **doc/OAI-PMH.md** | Repository | OAI-PMH harvesting verbs and usage | +| **GitHub Wiki** | External | Extended API documentation (linked from urlAPI.jsp) | + +**PhyloWS Endpoints:** +- `/phylows/study/**` - Study records (PhyloWSStudyController) +- `/phylows/tree/**` - Phylogenetic trees (PhyloWSTreeController) +- `/phylows/matrix/**` - Character matrices (PhyloWSMatrixController) +- `/phylows/taxon/**` - Taxonomic data (PhyloWSTaxonController) +- `/phylows/classification/**` - Classification data (PhyloWSClassificationController) + +**RSS/Atom Feed Endpoints:** +- `/rss.xml` - RSS 2.0 feed of recent studies and updates +- `/feed.xml` - Atom feed of recent studies and updates + +**Output Formats:** HTML, NeXML, NEXUS, RDF, RSS 1.0, RSS 2.0, Atom, JSON + +### Data Standards + +Documentation for phylogenetic data formats is distributed across in-app pages and external links: + +| Format | Coverage | Location | +|--------|----------|----------| +| **NEXUS** | Submission format | submitTutorial.jsp, Mesquite documentation | +| **NeXML** | Primary output format | doc/API.md, nexml.org (external) | +| **Newick** | Tree representation | Implicit in tree endpoints | +| **RDF/CDAO** | Semantic web output | doc/API.md, evolutionaryontology.org | +| **RSS 1.0** | Search result feeds | doc/API.md | +| **RSS 2.0** | Recent studies feed | /rss.xml endpoint | +| **Atom** | Recent studies feed | /feed.xml endpoint | + +**Data Preparation:** +- **submitTutorial.jsp** (`/submitTutorial.html`) - NEXUS file preparation using Mesquite +- YouTube instructional videos embedded in submitTutorial.jsp + +**Metadata Standards:** +- Dublin Core (dcterms) predicates for bibliographic metadata +- PRISM for publication dates +- TreeBASE-specific predicates (tb:) documented in API.md + +### Developer Resources + +Resources for contributors and developers: + +| Resource | Location | Description | +|----------|----------|-------------| +| **technology.jsp** | `/technology.html` | Architecture overview, technology stack diagram | +| **BUILDING.md** | `doc/development/` | Build prerequisites (Java 17, Maven), compilation steps | +| **DWR.md** | `doc/technical-notes/` | AJAX integration with Spring 5 compatibility | +| **UPGRADES.md** | `doc/technical-notes/` | Dependency upgrade history and migration notes | +| **GitHub Repository** | github.com/TreeBASE/treebase | Source code under BSD license | +| **GitHub Issues** | Linked from contact.jsp | Bug reports and feature requests | + +**Technology Stack (from technology.jsp):** +- Database: PostgreSQL +- ORM: Hibernate +- Framework: Spring +- File Parsing: Mesquite +- Tree Visualization: phylotree.js + +**Project Structure:** +- `treebase-core` - ORM API for PostgreSQL database access +- `treebase-web` - MVC web application with JSP/HTML GUI +- `oai-pmh_data_provider` - OAI-PMH interface functionality + +### Integration Guides + +Programmatic integration documentation: + +| Integration Type | Documentation | Key Information | +|-----------------|---------------|-----------------| +| **REST API Queries** | doc/API.md | PhyloWS URL patterns, CQL search syntax | +| **Metadata Harvesting** | doc/OAI-PMH.md | OAI-PMH verbs: Identify, ListRecords, GetRecord, etc. | +| **Data Download** | urlAPI.jsp | NEXUS, NeXML, JSON format parameters | +| **RSS Feeds** | doc/API.md, urlAPI.jsp | RSS 1.0 for search results and alerts | + +**URI Patterns (from urlAPI.jsp):** +- Study: `{purlBase}study/TB2:S{id}` +- Matrix: `{purlBase}matrix/TB2:M{id}` +- Tree: `{purlBase}tree/TB2:Tr{id}` + +**Search Examples (from API.md):** +- `/study/find?query=dcterms.contributor=Huelsenbeck` +- `/taxon/find?query=dcterms.title=="Homo sapiens"&format=rss1&recordSchema=tree` + +**Note:** No official client libraries or SDKs are currently provided. + +### Database Documentation + +Database and data model documentation: + +| Resource | Location | Description | +|----------|----------|-------------| +| **technology.jsp** | `/technology.html` | Data content overview, architecture diagram | +| **BUILDING.md** | `doc/development/` | Database connection configuration | +| **Hibernate Mappings** | `treebase-core` source | Entity relationships via ORM | + +**Data Model Overview (from technology.jsp):** +1. Studies contain bibliographic references to published phylogenetic research +2. Each study has one or more analyses; each analysis has steps associating matrices, trees, algorithms, and software +3. Character matrices contain taxon labels mapped to tree leaf nodes +4. Row segments link to specimen, tissue, or gene sequence metadata +5. Taxon names map to uBio NameBank and NCBI taxonomy + +**Key Entity Types:** +- Study, Citation, Analysis, AnalysisStep +- Matrix, Tree, TreeBlock +- Taxon, TaxonLabel, TaxonVariant + +**Note:** Formal data dictionary and entity-relationship diagrams are not currently published. + +### Deployment + +Deployment and operations documentation: + +| Resource | Location | Description | +|----------|----------|-------------| +| **BUILDING.md** | `doc/development/` | Prerequisites, compilation, WAR packaging | +| **DEPLOYING.md** | `doc/development/` | Tomcat configuration, JVM arguments, verification | +| **UPGRADES.md** | `doc/technical-notes/` | Dependency versions, security fixes, migration paths | +| **DOCKER.md** | Repository root | Docker deployment option | +| **docker-compose.yml** | Repository root | Container orchestration | + +**System Requirements (from BUILDING.md/DEPLOYING.md):** +- Java 17 LTS +- Maven 3.8+ +- PostgreSQL database +- Tomcat 7+ servlet container +- Headless Mesquite for file parsing + +**Build Artifacts:** +- `treebase-web.war` (~59MB) - Main web application +- `data_provider_web.war` - OAI-PMH interface + +**Configuration Files:** +- `jdbc.properties` - Database connection +- `context.xml` - Tomcat context with database and Mesquite paths + +## External Documentation + +*Links to existing technical documentation* + +- [API.md](../API.md) - PhyloWS API documentation +- [OAI-PMH.md](../OAI-PMH.md) - OAI-PMH harvesting interface +- [Development Documentation](../development/) - Build and deployment guides +- [Technical Notes](../technical-notes/) - Implementation details + +## Pages to Account For + +Complete inventory of JSP pages and controllers related to technical documentation: + +| Page | URL Pattern | Controller | Status | +|------|-------------|------------|--------| +| urlAPI.jsp | /urlAPI.html | filenameController | ✅ Active | +| technology.jsp | /technology.html | filenameController | ✅ Active | +| about.jsp | /about.html | filenameController | ✅ Active | +| contact.jsp | /contact.html | filenameController | ✅ Active | +| submitTutorial.jsp | /submitTutorial.html | filenameController | ✅ Active | +| help.jsp | /help.html | helpController | ✅ Active | +| sitemap.jsp | /sitemap.xml | sitemapController | ✅ Active | + +**PhyloWS Controllers:** + +| Controller | URL Pattern | Description | +|------------|-------------|-------------| +| PhyloWSStudyController | /phylows/study/** | Study data API | +| PhyloWSTreeController | /phylows/tree/** | Tree data API | +| PhyloWSMatrixController | /phylows/matrix/** | Matrix data API | +| PhyloWSTaxonController | /phylows/taxon/** | Taxon data API | +| PhyloWSClassificationController | /phylows/classification/** | Classification API | + +**OAI-PMH Interface:** + +| Endpoint | URL Pattern | Description | +|----------|-------------|-------------| +| OAI-PMH Provider | /top/oai | Metadata harvesting service | + +**RSS/Atom Feed Endpoints:** + +| Endpoint | URL Pattern | Output Format | Description | +|----------|-------------|---------------|-------------| +| RSS Feed | /rss.xml | RSS 2.0 | Recent studies and updates feed | +| Atom Feed | /feed.xml | Atom | Recent studies and updates feed | + +**JSON Output Pages (for API responses):** + +| Page | Purpose | +|------|---------| +| json/studyToJSON.jsp | Study JSON serialization | +| json/treeToJSON.jsp | Tree JSON serialization | +| json/matrixToJSON.jsp | Matrix JSON serialization | +| json/taxonToJSON.jsp | Taxon JSON serialization | +| anyObjectAsRDF.jsp | RDF output generation | +| searchResultsAsRDF.jsp | Search results as RDF | + +## Wireframe Notes + +*To be completed in future PR* + +## Open Questions + +- Where should API documentation live (in-app vs external)? +- What interactive API features are needed (playground, testing)? +- How should code examples be maintained? +- What versioning strategy for API documentation? diff --git a/doc/v3/user-story-07-governance.md b/doc/v3/user-story-07-governance.md new file mode 100644 index 00000000..70eb8e3f --- /dev/null +++ b/doc/v3/user-story-07-governance.md @@ -0,0 +1,185 @@ +# User Story 7: Governance + +## Overview + +**As a** governance stakeholder, +**I want to** learn about the project, its history, contributors, and role in phylogenetic data +management, +**So that** I can understand TreeBASE's position in the scientific data ecosystem and make +informed decisions about its future. + +## User Types + +- Funding agency representatives +- Institutional administrators +- Scientific advisory board members +- Policy makers +- Potential partners and collaborators +- Science journalists and communicators + +## Current Pages + +The following pages in the current TreeBASE web application relate to this user story: + +- [x] About page (`/about.html`) - Background information, history, funding, governance, and related resources +- [ ] History/timeline page - *Not a separate page; history content is included in the About page* +- [x] Contributors/team page (`/people.html`) - Current and past contributors with photos and roles +- [x] Mission statement - *Included in About page and Home page* +- [x] Data usage policies (`/dataMan.html`) - NSF Data Management Plan covering submission, integrity, standards, and persistence +- [ ] Terms of service - *Not currently available as a dedicated page* +- [ ] Privacy policy - *Not currently available as a dedicated page* +- [x] Contact information (`/contact.html`) - Help desk email and bug reporting via GitHub issues +- [ ] Governance structure - *Basic information included in About page; no dedicated governance page* +- [x] Funding acknowledgments - *Included in About page with NSF grant numbers* +- [x] Partnerships page (`/partnership.html`) - Current and historical institutional partners +- [x] References/Citations page (`/reference.html`) - How to cite TreeBASE publications +- [x] Technology page (`/technology.html`) - Implementation details and source code information +- [x] Journals page (`/journal.html`) - Partner journals and publication policies + +## Navigation Flow + +Users navigate to governance-related pages primarily through the left sidebar menu available on the home page and all main pages. The current navigation structure is: + +```mermaid +flowchart TD + Home[Home /home.html] --> Sidebar[Left Sidebar Navigation] + + Sidebar --> Search[Search TreeBASE /search/studySearch.html] + Sidebar --> Submit[Submit Data /user/processUser.html] + + Sidebar --> Documentation + Documentation --> Technology[Technology /technology.html] + Documentation --> Tutorial[Submit Tutorial /submitTutorial.html] + Documentation --> API[Data Access/API /urlAPI.html] + + Sidebar --> About + About --> Overview[Overview /about.html] + About --> People[People /people.html] + About --> Partnerships[Partnerships /partnership.html] + About --> References[References /reference.html] + + Sidebar --> DataMan[NSF Data Management /dataMan.html] + Sidebar --> Journals[Journals /journal.html] + Sidebar --> Contact[Contact /contact.html] + + Sidebar --> Footer[Footer Links] + Footer --> Mendeley[Mendeley Group] + Footer --> Twitter[Twitter @TreeBASE] + Footer --> DCI[Data Citation Index] +``` + +**External Links in Footer:** +- Mendeley group for TreeBASE publications +- Twitter @TreeBASE +- Data Citation Index badge + +## Governance Information + +### About TreeBASE + +**Coverage:** `/about.html` and `/home.html` + +- What is TreeBASE - ✅ Covered in About page ("Background" section) and Home page welcome text +- Mission statement - ✅ Covered in About page and Home page (TreeBASE is a repository of phylogenetic information) +- Vision for the future - ⚠️ Partially covered; About page mentions current version but no explicit future vision +- Value proposition - ✅ Covered in About page (applications list) and Home page (new features list) + +### History + +**Coverage:** `/about.html` ("History, Funding, and Governance" section) + +- Timeline of development - ✅ Covered (prototype 1994, redevelopment with CIPRES, v2.0 March 2010) +- Major milestones - ✅ Covered (NSF grants, institutional hosts over time) +- Evolution of the platform - ✅ Covered (mentions previous hosts: NESCent, Yale Peabody, SDSC, UB, Harvard, Leiden, UC Davis) +- Historical context in phylogenetics - ✅ Covered (references to Sanderson et al., Piel et al. publications) + +### Team and Contributors + +**Coverage:** `/people.html` + +- Current team members - ✅ Covered with photos, names, and roles +- Advisory board - ❌ Not explicitly listed +- Past contributors - ✅ Covered ("Past developers" section) +- Institutional affiliations - ✅ Covered (institutions shown with contributor entries) + +### Governance Structure + +**Coverage:** `/about.html`, `/partnership.html`, `/journal.html` + +- Decision-making processes - ⚠️ Partially covered; mentions Phyloinformatics Research Foundation governance +- Organizational structure - ✅ Covered in About page (Phyloinformatics Research Foundation, Inc.) +- Partner institutions - ✅ Covered in Partnerships page (Naturalis, NESCent, CIPRES, Dryad) +- Community involvement - ⚠️ Limited coverage; GitHub issues for bug reports mentioned in Contact +- Relationship with journals - ✅ Covered in Journals page (`/journal.html`) - lists partner journals that recommend/require TreeBASE submission, provides PhyloWS URLs for each journal's studies + +### Policies + +**Coverage:** `/dataMan.html`, `/submitTutorial.html` + +- Data submission policies - ✅ Covered in NSF Data Management page and Submit Tutorial +- Data usage and licensing - ✅ Covered in NSF Data Management page (public domain, no restrictions on reuse) +- Terms of service - ❌ No dedicated page +- Privacy policy - ❌ No dedicated page +- Code of conduct - ❌ No dedicated page + +### Impact and Metrics + +**Coverage:** `/home.html`, `/reference.html` + +- Usage statistics - ⚠️ Partially covered; Home page shows counts (4,076 publications, 8,777 authors, etc. as of April 2014 - needs update) +- Citation information - ✅ Covered in References page (how to cite TreeBASE) +- Data holdings summary - ✅ Covered in Home page (matrices, trees, taxon labels counts) +- Community reach - ⚠️ Limited; Twitter and Mendeley links in sidebar, Data Citation Index badge + +### Funding and Support + +**Coverage:** `/about.html`, `/partnership.html` + +- Current funders - ⚠️ Limited; current host (Naturalis) mentioned but no current funding details +- Historical funding - ✅ Covered in About page (NSF grants DEB 9318325, EF 0331654) +- How to support TreeBASE - ❌ Not covered +- Institutional partnerships - ✅ Covered in Partnerships page + +### Contact and Engagement + +**Coverage:** `/contact.html`, sidebar links, GitHub + +- Contact information - ✅ Covered in Contact page (help@treebase.org) +- Social media presence - ✅ Twitter link in sidebar +- Community forums - ✅ Available via GitHub: + - Issues/bugs: https://github.com/TreeBASE/treebase/issues + - Discussions: https://github.com/TreeBASE/treebase/discussions + +## Pages to Account For + +The following is a complete inventory of pages related to governance: + +| Page | URL Pattern | Status | Notes | +|------|-------------|--------|-------| +| Home | `/home.html` | ✅ Exists | Welcome message, mission overview, recent studies | +| About/Overview | `/about.html` | ✅ Exists | Background, history, funding, governance, related resources | +| People/Team | `/people.html` | ✅ Exists | Current and past contributors with photos | +| Partnerships | `/partnership.html` | ✅ Exists | Institutional partners (Naturalis, NESCent, CIPRES, Dryad) | +| References | `/reference.html` | ✅ Exists | How to cite TreeBASE | +| Technology | `/technology.html` | ✅ Exists | Implementation stack, source code links | +| Submit Tutorial | `/submitTutorial.html` | ✅ Exists | Submission process documentation | +| Data Access/API | `/urlAPI.html` | ✅ Exists | Web service and programmatic access | +| NSF Data Management | `/dataMan.html` | ✅ Exists | Data management plan details | +| Journals | `/journal.html` | ✅ Exists | Partner journals requiring TreeBASE submissions | +| Contact | `/contact.html` | ✅ Exists | Help desk email, GitHub issue tracker | +| Terms of Service | N/A | ❌ Missing | No dedicated page | +| Privacy Policy | N/A | ❌ Missing | No dedicated page | +| Governance Structure | N/A | ❌ Missing | Basic info in About page only | +| Usage Statistics | N/A | ❌ Missing | Some stats on home page, no dedicated page | +| Impact Metrics | N/A | ❌ Missing | No dedicated page | + +## Wireframe Notes + +*To be completed in future PR* + +## Open Questions + +- What governance information should be prominently featured? +- How should historical information be presented? +- What metrics should be publicly displayed? +- How should contact/support requests be handled? diff --git a/treebase-core/db/schema/patches/0009_create-password-reset-token-table.sql b/treebase-core/db/schema/patches/0009_create-password-reset-token-table.sql deleted file mode 100644 index 52644dc7..00000000 --- a/treebase-core/db/schema/patches/0009_create-password-reset-token-table.sql +++ /dev/null @@ -1,41 +0,0 @@ --- --- Schema patch 0009: Create password_reset_token table --- --- This table stores secure tokens for password reset functionality. --- Tokens expire after a configurable time period and can only be used once. --- - --- Create sequence for token_id -CREATE SEQUENCE password_reset_token_id_sequence - START WITH 1 - INCREMENT BY 1 - NO MAXVALUE - NO MINVALUE - CACHE 1; - --- Create password_reset_token table -CREATE TABLE password_reset_token ( - token_id bigint DEFAULT nextval('password_reset_token_id_sequence'::regclass) NOT NULL, - token character varying(100) NOT NULL, - user_id bigint NOT NULL, - expiry_date timestamp without time zone NOT NULL, - used boolean DEFAULT false NOT NULL -); - --- Add primary key constraint -ALTER TABLE ONLY password_reset_token - ADD CONSTRAINT password_reset_token_pkey PRIMARY KEY (token_id); - --- Add unique constraint on token -ALTER TABLE ONLY password_reset_token - ADD CONSTRAINT password_reset_token_token_key UNIQUE (token); - --- Add foreign key constraint to user table -ALTER TABLE ONLY password_reset_token - ADD CONSTRAINT password_reset_token_user_fk FOREIGN KEY (user_id) REFERENCES "user"(user_id) ON DELETE CASCADE; - --- Create index on user_id for faster lookups -CREATE INDEX password_reset_token_user_idx ON password_reset_token USING btree (user_id); - --- Create index on expiry_date for faster cleanup queries -CREATE INDEX password_reset_token_expiry_idx ON password_reset_token USING btree (expiry_date);