Evaluations

An Evaluation is essentially a container that organizes and manages the submission, assessment, and scoring of data, models, or other research artifacts. It allows teams to set up challenges where participants contribute their work, and those contributions can be systematically reviewed or scored.

This tutorial will walk you through the basics of working with Evaluations using the Synapse Python client.

Tutorial Purpose

In this tutorial you will:

1. Create and update an Evaluation on Synapse
2. Update the ACL (Access Control List) of an Evaluation on Synapse
3. Retrieve and delete all Evaluations from a given Project

Prerequisites

• You have completed the Project tutorial, or have an existing Project on Synapse to work from
• You have a Synapse user or team ID to share the evaluation with

1. Create and update an Evaluation on Synapse

In this first part, we'll be showing you how to interact with an Evaluation object as well as introducing you to its two core functionalities store() and get().

from synapseclient import Synapse
from synapseclient.models import Evaluation, Project

syn = Synapse()
syn.login()

# REQUIRED: Set this to the Synapse user ID or team ID you want to grant permissions to
# Do NOT leave this as None - the script will not work properly
PRINCIPAL_ID = None  # Replace with actual user/team ID

# Retrieve the Project where your Evaluation will be stored
project = Project(name="My uniquely named project about Alzheimer's Disease").get()
project_id = project.id

print(f"Working within Project: {project_id}")

# Create a new Evaluation object
evaluation = Evaluation(
    name="My Challenge Evaluation for Study ABC - Round 1",
    description="Evaluation for my data challenge",
    content_source=project_id,
    submission_instructions_message="Submit CSV files only",
    submission_receipt_message="Thank you for your submission!",
)

# Create the Evaluation on Synapse
evaluation.store()

print("Evaluation has been created with the following name and description:")
print(evaluation.name)
print(evaluation.description)

# Update the Evaluation object's name and description
evaluation.name = "My Challenge Evaluation for Study XYZ - Round 1"
evaluation.description = "Updated description for my evaluation"

# Update the Evaluation on Synapse
evaluation.store()

print("Evaluation has been updated with the following name and description:")
print(evaluation.name)
print(evaluation.description)

2. Update the ACL of an Evaluation on Synapse

Like Synapse entities, Evaluations have ACLs that can be used to control who has access to your evaluations and what level of access they have. Updating the ACL of an Evaluation object is slightly different from updating other Evaluation components, because the ACL is not an attribute of the Evaluation object. Let's see an example of how this looks:

# Update the Evaluation's ACL on Synapse by adding a new user
assert (
    PRINCIPAL_ID is not None
), "PRINCIPAL_ID must be set to the Synapse user ID or team ID you want to grant permissions to."

evaluation.update_acl(principal_id=PRINCIPAL_ID, access_type=["READ", "SUBMIT"])

# Get the Evaluation's ACL to confirm the update
acl = evaluation.get_acl()
print("The following ACL has been retrieved from Synapse:")
print(acl)

You can also remove principals from an ACL by simply feeding update_acl an empty list for the access_type argument, like so:

# Now let's remove the user we just added from the Evaluation's ACL
evaluation.update_acl(principal_id=PRINCIPAL_ID, access_type=[])

3. Retrieve and delete all Evaluations from a given Project

Now we will show how you can retrieve lists of Evaluation objects, rather than retrieving them one-by-one with get(). This is a powerful tool if you want to perform the same action on all the evaluations in a given project, for example, like what we're about to do here:

# Finally let's retrieve all Evaluations stored within this project, including the one we just created
evaluations_list = Evaluation.get_evaluations_by_project(project_id)

# Let's delete the evaluation we created for this tutorial, and any other evaluations in this project (uncomment below to enable deletion)
# for evaluation_to_delete in evaluations_list:
#     print(f"Deleting evaluation: {evaluation_to_delete.name}")
#     evaluation_to_delete.delete()

Source code for this tutorial

Click to show me

"""
Here is where you'll find the code for the Evaluation tutorial.
"""

from synapseclient import Synapse
from synapseclient.models import Evaluation, Project

syn = Synapse()
syn.login()

# REQUIRED: Set this to the Synapse user ID or team ID you want to grant permissions to
# Do NOT leave this as None - the script will not work properly
PRINCIPAL_ID = None  # Replace with actual user/team ID

# Retrieve the Project where your Evaluation will be stored
project = Project(name="My uniquely named project about Alzheimer's Disease").get()
project_id = project.id

print(f"Working within Project: {project_id}")

# Create a new Evaluation object
evaluation = Evaluation(
    name="My Challenge Evaluation for Study ABC - Round 1",
    description="Evaluation for my data challenge",
    content_source=project_id,
    submission_instructions_message="Submit CSV files only",
    submission_receipt_message="Thank you for your submission!",
)

# Create the Evaluation on Synapse
evaluation.store()

print("Evaluation has been created with the following name and description:")
print(evaluation.name)
print(evaluation.description)

# Update the Evaluation object's name and description
evaluation.name = "My Challenge Evaluation for Study XYZ - Round 1"
evaluation.description = "Updated description for my evaluation"

# Update the Evaluation on Synapse
evaluation.store()

print("Evaluation has been updated with the following name and description:")
print(evaluation.name)
print(evaluation.description)

# Confirm what's in Synapse matches the evaluation stored
from_synapse = Evaluation(id=evaluation.id).get()

print("The following evaluation has been retrieved from Synapse:")
print(from_synapse)

# Update the Evaluation's ACL on Synapse by adding a new user
assert (
    PRINCIPAL_ID is not None
), "PRINCIPAL_ID must be set to the Synapse user ID or team ID you want to grant permissions to."

evaluation.update_acl(principal_id=PRINCIPAL_ID, access_type=["READ", "SUBMIT"])

# Get the Evaluation's ACL to confirm the update
acl = evaluation.get_acl()
print("The following ACL has been retrieved from Synapse:")
print(acl)

# Now let's remove the user we just added from the Evaluation's ACL
evaluation.update_acl(principal_id=PRINCIPAL_ID, access_type=[])

# Finally let's retrieve all Evaluations stored within this project, including the one we just created
evaluations_list = Evaluation.get_evaluations_by_project(project_id)

# Let's delete the evaluation we created for this tutorial, and any other evaluations in this project (uncomment below to enable deletion)
# for evaluation_to_delete in evaluations_list:
#     print(f"Deleting evaluation: {evaluation_to_delete.name}")
#     evaluation_to_delete.delete()

References

• Evaluation
• Project
• syn.login