Apr 20, 2023

Public workspaceEvaluating Collaboratory Cultures

This protocol is a draft, published without a DOI.
  • 1UVA School of Data Science
M Fox
Open access
Protocol Citation: M Fox, Phil Bourne 2023. Evaluating Collaboratory Cultures. protocols.io https://protocols.io/view/evaluating-collaboratory-cultures-cs6jwhcn
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License,  which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: In development
We are still developing and optimizing this protocol
Created: April 20, 2023
Last Modified: April 20, 2023
Protocol Integer ID: 80811
Keywords: interdisciplinary, open science, collaboratory, research culture
Funders Acknowledgement:
Jefferson Trust
Grant ID: FAAJ3302
Disclaimer
DISCLAIMER – FOR INFORMATIONAL PURPOSES ONLY; USE AT YOUR OWN RISK

The protocol content here is for informational purposes only and does not constitute legal, medical, clinical, or safety advice, or otherwise; content added to protocols.io is not peer reviewed and may not have undergone a formal approval of any kind. Information presented in this protocol should not substitute for independent professional judgment, advice, diagnosis, or treatment. Any action you take or refrain from taking using or relying upon the information presented here is strictly at your own risk. You agree that neither the Company nor any of the authors, contributors, administrators, or anyone else associated with protocols.io, can be held responsible for your use of the information contained in or linked to this protocol or any of our Sites/Apps and Services.
Abstract
The disciplinary silos in Higher Education were organized around defined fields of study in order to facilitate student instruction—law students receive their education in the Law College, medical students at the School of Medicine, etc. But these traditional arrangements are not appropriate for 21st century science, which is organized around research relationships in order to optimize diversity and accelerate innovation. The collaboratory structure, such as the School of Data Science (SDS) Collaboratories, strikes a balance between UVA’s pride in tradition and its Great and Good strategic plan to celebrate its tricentennial as the world’s best public university.
But does this innovative structure provide a measurable improvement to tradition? How do collaboratory practices facilitate future trends in science, such as convergence research with international teams and cross-functional capabilities? How can UVA students, faculty, trustees, and administration use insights into their collaboratory functioning to improve decisions? Looking forward to 2030, how can UVA leverage its unique advantage in collaboratory expertise toward becoming the nation’s premier Collaboratory Research Center with training, certification, and services for industry, academia, civil society, and government?
The topic of collaboration as a research subject has been well-studied in most fields, and some metadata about collaboratories is well-documented (e.g., h-index, bibliometrics, award amounts, recipient demographics, etc.); however, metrics and reference data that can be used to accurately assess the functioning of the collaboratory unit have yet to be established and modeled. Indeed, there is not a single published dataset or evidence-based best practice to inform the design and/or operation of the research collaboratory. In short, the collaboratory as a subject of scientific research remains unexamined. This systemic lack of performance metrics results in persistent shortcomings that pervade every level of decision-making—from program selection by graduate school applicants to strategic planning within the Office of the President.
Image Attribution
CC-BY
Landscaping | Facilitation | Framework Development & Testing
Landscaping | Facilitation | Framework Development & Testing
Phase I: Landscaping
This phase of the study is focused on document review to develop benchmarks and semi-structured interviews to examine the assumptions, values, practices, challenges, needs, perceptions, and services that constitute UVA’s research landscape.
Document Review
Published policies and documents relating to interdisciplinary research initiatives, guidelines, and services will be found posted online in UVA department sites and at the UVA Library.Unpublished assessments, reports, meeting notes, strategic plans, and other documents will be found offline and may be identified and produced upon request by individuals involved in the decision-making process.
Semi-structured Interviews
Findings from the Document Review process will inform the specific set of questions posed to members of the UVA research community. Likewise, the responses to the interviews will inform the development of baseline metrics to capture the current state of interdisciplinary values and commitments on campus, and then measure progress toward community-defined goals going forward. Nonetheless, a preliminary line of inquiry has been prepared to address the practice and perception of interdisciplinarity on campus.
Phase II: Facilitation

Phase III: Framework Development & Testing
The first Collaboratory Cultures framework that we will test is the FAIR + OA Framework. This approach applies FAIR data science principles to the Open Access movement and expresses the framework output as a series context-specific of use cases and applications. At this time, analysis of semi-structured interviews and surveys to be conducted in Phases I-II will be required to develop framework outputs.In the future, we plan to develop tooling that will automate context-specific use case and application outputs, similar to the bottom two rows of the example rubric above.
Surveys

Assign a value to each practice (No=0 | Yes=1)

Communication practices

  • Establish and explain agile communication practices early
  • Co-create the communication platforms and tools strategy
  • Define the purpose of the collaboration
  • Describe, debate, and refine purpose in the first team meeting
  • Elicit co-creation/participatory design input early and often
  • Assign roles and clarify responsibilities for all collaboratory members involved in the project lifecycle, including principal investigators and team leads
  • Outline benchmarks of success (i.e., project milestones and schedule of deliverables)
  • Introduce collaboration tools and how they relate to the purpose of the project, such as communication platforms and meeting schedules
  • Provide collaboratory orientation (team introductions, SOP, tool and method training info)

Video conference practices

  • Optimize productivity by adopting agile techniques and anti-fatigue good practices
oDisable mirror cam function during meetings
oDisable cams during presentation of meeting materials
oReduce cam time as much as possible to pull attention to agenda topics and reduce bandwidth issues
  • Prioritize inclusivity by normalizing privacy practices, supporting cultural differences, and offsetting nonverbal cues
oNever ask a participant to turn on their cam if they have chosen to participate off camera
oEnforce the use of the “Raise Hand” function to support uninterrupted speech sans nonverbal cues
  • Provide secure, stable links to project meetings (including agenda) for asynchronous participation


Participation best practices

  • Always ask for clarification when you are uncertain
  • Offer your own disciplinary input to help refine or expand issues
  • Respect and encourage a diversity of opinions, backgrounds, and experiences
  • Avoid jargon when a synonym can be used
  • When jargon is unavoidable, define terms
  • Address conflicting ideas or approaches as they arise and employ co-creation techniques in the resolution process


Communicating multidisciplinary concepts

  • Schedule enough time in the meeting agenda to introduce new terms and concepts
  • Practice iterative review to raise competency across the entire collaboratory
  • Issue a project handbook or SOP and add terms and definitions, as necessary


Collaboratory tools

  • Document work on shared docs that allow multiple users to take notes, edit, and comment in real time.
  • Allow team members to schedule meetings, set deadlines, assign time-sensitive tasks, and track project milestones on a shared calendar.
  • Use instant messaging when it is the prescribed method of communication in the SOP (avoid using IM as an ad hoc communication tool to prevent fatigue)
  • Use data repositories to track version control, multiple threads, and updates.
  • Practice project archiving to establish a stable record of project assets and to track progress via reflexive practice.
  • Establish reproducible and executable software configuration practices, commonly referred to as containerization, bundles together software, configuration files, dependent versions, and data so that projects can be reproduced, regardless of future changes in operating systems and software versions.
  • Preserve data assets with computational notebooks and tutorials that combine code, results, and descriptive text into a computational narrative.
  • Use scientific discovery platforms to enable collaboration with big data assets.


Data management best practices

  • Make data as FAIR as possible.
  • Revisit the data management plan frequently during the project and make changes as necessary.
  • Consider legal, ethical, and cultural obligations when drafting sharing policies.
  • Implement a stewardship best practices strategy and reflexive methodology at the project outset.


Code writing practices

  • Good practices for creating sustainable, accessible codes:
oProvide a step-by-step user manual for tools whenever possible
oAlways maintain an up-to-date Readme file where latest updates and requirements are listed
oProvide high-level comments at the beginning of each file and throughout the code as needed
oFollow consistent naming convention across your codes
oDo version-control
  • Data curation:
oAutomate as many of the processes involved in data access, storage, and reformatting as possible.
oKeep separate copies of the original (raw) data and the curated data
  • Data analysis:
oResearch and employ common, successful analysis methods.Do not reinvent the wheel.
oMap your method to the research questions you are trying to answer.Do not try to fit your method to an application.
oMake a tutorial-style document that explains your analytical method in simple language.
  • Data visualization practices:
oUse visual graphics to communicate results with collaborators.
oUse high quality formats to produce images.
oAutomate data visualization as much as possible.
oAim for users being only a single click away from reproducing everything.
  • Build stand-alone tools:
oIf possible and where appropriate, build a Graphical User Interface (GUI) that allows your collaborators to tweak parameters and apply their expertise to parameter evaluation and exploratory analyses.
oWhen appropriate, build add-on packages and libraries, or even lean software tools.


Participatory design practices

  • Create a flexible-by-design, agile management framework that can accommodate variable scope and unanticipated results.
  • Specify the distinct contribution that each collaborator has to offer to their field.
  • Identify inclusive objectives and/or outputs that allow each contributor to advance their own professional goals and research agendas.
  • Specify how the results of collaborative research, including data science methods, will ultimately be evaluated by disciplinary experts.
  • Ensure that individual contributions will contribute to the research portfolio of the data science practitioners, and vice versa.
  • Revise and improve the research plan whenever new collaborators are onboarded.


Roles and responsibilities

  • Co-create a project management plan with milestones and deadlines that lead to clearly defined project goals by assigning roles and tasks according to the strengths and interests of each team member.
  • Invest in the personal contribution of individual team members by working one-on-one to design and refine assigned tasks.
  • Check-in regularly to ensure that each individual is on track with the tools and resources they need to succeed, and adjusting the plan as needed.