Evaluation

synapseclient.evaluation ¶

Evaluations¶

An Evaluation object represents a collection of Synapse Entities that will be processed in a particular way. This could mean scoring Entries in a challenge or executing a processing pipeline.

Imports and authentication:

from synapseclient import Evaluation, Submission, SubmissionStatus, Synapse

syn = Synapse()
syn.login()

Evaluations can be retrieved by ID:

evaluation = syn.getEvaluation(1901877)

Like entities, evaluations are access controlled via ACLs. The synapseclient.Synapse.getPermissions and synapseclient.Synapse.setPermissions methods work for evaluations:

access = syn.getPermissions(evaluation, user_id)

The synapseclient.Synapse.submit method returns a Submission object:

entity = syn.get(synapse_id)
submission = syn.submit(evaluation, entity, name='My Data', team='My Team')

The getSubmissionStatus function can then be used to check the SubmissionStatus of the submission:

status = syn.getSubmissionStatus(submission)

The status of a submission may be

INVALID the submitted entity is in the wrong format
SCORED in the context of a challenge or competition
OPEN indicating processing has not completed
CLOSED indicating processing has completed

SubmissionStatus objects can be updated, usually by changing the status and score fields, and stored back to Synapse using synapseclient.Synapse.store:

status.score = 0.99
status.status = 'SCORED'
status = syn.store(status)

See:

Classes¶

Evaluation ¶

Bases: DictObject

An Evaluation Submission queue, allowing submissions, retrieval and scoring.

WARNING - This class is deprecated and will no longer be maintained. Please use the Evaluation model from synapseclient.models.evaluation instead.

PARAMETER	DESCRIPTION
`name`	Name of the evaluation
`description`	A short description of the evaluation
`contentSource`	Synapse Project associated with the evaluation
`submissionReceiptMessage`	Message to display to users upon submission
`submissionInstructionsMessage`	Message to display to users detailing acceptable formatting for submissions.

Create and store an Evaluation (DEPRECATED)

To create an Evaluation and store it in Synapse:

import synapseclient
from synapseclient import Evaluation

## Initialize a Synapse object & authenticate
syn = synapseclient.Synapse()
syn.login()

evaluation = syn.store(Evaluation(
    name="Q1 Final",
    description="Predict progression of MMSE scores for final scoring",
    contentSource="syn12345"))

The contentSource field links the evaluation to its Project (or, really, any synapse ID, but sticking to projects is a good idea).

Create and store an Evaluation (NEW METHOD)

from synapseclient import Synapse
from synapseclient.models import Evaluation

# Create client and login
syn = Synapse()
syn.login()

evaluation = Evaluation(name="My Challenge Evaluation",
                        description="Description of my challenge evaluation",
                        content_source="syn12345",
                        submission_instructions_message="Please submit your entries in the format described here: ...",
                        submission_receipt_message="Thank you for your submission! Your entry has been received.")

evaluation.store()

Retrieve an Evaluation multiple ways (DEPRECATED)

Evaluations can be retrieved from Synapse by ID, content source (associated entity), or name:

# Retrieve evaluation by ID
evaluation = syn.getEvaluation(9999999)

# Retrieve evaluation by name
evaluation = syn.getEvaluationByName('Foo Challenge Question 1')

# Retrieve evaluation by content source (associated entity)
evaluation = syn.getEvaluationByContentSource('syn12345')

Retrieve an Evaluation multiple ways (NEW METHOD)

from synapseclient import Synapse
from synapseclient.models import Evaluation

# Create client and login
syn = Synapse()
syn.login()

# Retrieve evaluation by ID
evaluation = Evaluation(id=9999999).get()

# Retrieve evaluation by name
evaluation = Evaluation(name='Foo Challenge Question 1').get()

# Retrieve all evaluations by content source (associated entity)
evaluation = Evaluation.get_evaluations_by_project(id='syn12345')

# Retrieve all evaluations the active user can submit to
evaluations_generator = Evaluation.get_available_evaluations()

# Retrieve all evaluations the active user has READ/VIEW access to
evaluations_generator = Evaluation.get_all_evaluations()

# Retrieve all evaluations the active user has UPDATE/EDIT access to
evaluations_generator = Evaluation.get_all_evaluations(access_type=['UPDATE'])

Source code in synapseclient/evaluation.py

@deprecated(
    version="4.9.0",
    reason="To be removed in 5.0.0. "
    "Use the Evaluation model from synapseclient.models.evaluation instead.",
)
class Evaluation(DictObject):
    """
    An Evaluation Submission queue, allowing submissions, retrieval and scoring.

    WARNING - This class is deprecated and will no longer be maintained. Please use the Evaluation model from synapseclient.models.evaluation instead.

    Arguments:
        name: Name of the evaluation
        description: A short description of the evaluation
        contentSource: Synapse Project associated with the evaluation
        submissionReceiptMessage: Message to display to users upon submission
        submissionInstructionsMessage: Message to display to users detailing acceptable formatting for submissions.

    Example: Create and store an Evaluation (DEPRECATED)
        To create an [Evaluation](https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/Evaluation.html)
        and store it in Synapse:

            import synapseclient
            from synapseclient import Evaluation

            ## Initialize a Synapse object & authenticate
            syn = synapseclient.Synapse()
            syn.login()

            evaluation = syn.store(Evaluation(
                name="Q1 Final",
                description="Predict progression of MMSE scores for final scoring",
                contentSource="syn12345"))

    The *contentSource* field links the evaluation to its [Project][synapseclient.entity.Project]
    (or, really, any synapse ID, but sticking to projects is a good idea).

    Example: Create and store an Evaluation (NEW METHOD)
        &nbsp;
        ```python
        from synapseclient import Synapse
        from synapseclient.models import Evaluation

        # Create client and login
        syn = Synapse()
        syn.login()

        evaluation = Evaluation(name="My Challenge Evaluation",
                                description="Description of my challenge evaluation",
                                content_source="syn12345",
                                submission_instructions_message="Please submit your entries in the format described here: ...",
                                submission_receipt_message="Thank you for your submission! Your entry has been received.")

        evaluation.store()
        ```

    Example: Retrieve an Evaluation multiple ways (DEPRECATED)
        [Evaluations](https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/Evaluation.html)
        can be retrieved from Synapse by ID, content source (associated entity), or name:

            # Retrieve evaluation by ID
            evaluation = syn.getEvaluation(9999999)

            # Retrieve evaluation by name
            evaluation = syn.getEvaluationByName('Foo Challenge Question 1')

            # Retrieve evaluation by content source (associated entity)
            evaluation = syn.getEvaluationByContentSource('syn12345')

    Example: Retrieve an Evaluation multiple ways (NEW METHOD)
        &nbsp;
        ```python
        from synapseclient import Synapse
        from synapseclient.models import Evaluation

        # Create client and login
        syn = Synapse()
        syn.login()

        # Retrieve evaluation by ID
        evaluation = Evaluation(id=9999999).get()

        # Retrieve evaluation by name
        evaluation = Evaluation(name='Foo Challenge Question 1').get()

        # Retrieve all evaluations by content source (associated entity)
        evaluation = Evaluation.get_evaluations_by_project(id='syn12345')

        # Retrieve all evaluations the active user can submit to
        evaluations_generator = Evaluation.get_available_evaluations()

        # Retrieve all evaluations the active user has READ/VIEW access to
        evaluations_generator = Evaluation.get_all_evaluations()

        # Retrieve all evaluations the active user has UPDATE/EDIT access to
        evaluations_generator = Evaluation.get_all_evaluations(access_type=['UPDATE'])
        ```
    """

    @classmethod
    def getByNameURI(cls, name: str):
        quoted_name = urllib_urlparse.quote(name)
        return f"/evaluation/name/{quoted_name}"

    @classmethod
    def getURI(cls, id: Union[str, int]):
        return f"/evaluation/{id}"

    def __init__(self, **kwargs):
        kwargs["contentSource"] = kwargs.get("contentSource", "")
        if not kwargs["contentSource"].startswith(
            "syn"
        ):  # Verify that synapse Id given
            raise ValueError(
                'The "contentSource" parameter must be specified as a Synapse'
                " Entity when creating an Evaluation"
            )
        super(Evaluation, self).__init__(kwargs)

    def postURI(self):
        return "/evaluation"

    def putURI(self):
        return f"/evaluation/{self.id}"

    def deleteURI(self):
        return f"/evaluation/{self.id}"

    def getACLURI(self):
        return f"/evaluation/{self.id}/acl"

    def putACLURI(self):
        return "/evaluation/acl"

Submission ¶

Bases: DictObject

Builds a Synapse submission object.

PARAMETER	DESCRIPTION
`name`	Name of submission
`entityId`	Synapse ID of the Entity to submit
`evaluationId`	ID of the Evaluation to which the Entity is to be submitted
`versionNumber`	Version number of the submitted Entity
`submitterAlias`	A pseudonym or team name for a challenge entry

Source code in synapseclient/evaluation.py

class Submission(DictObject):
    """
    Builds a Synapse submission object.

    Arguments:
        name: Name of submission
        entityId: Synapse ID of the Entity to submit
        evaluationId: ID of the Evaluation to which the Entity is to be submitted
        versionNumber: Version number of the submitted Entity
        submitterAlias: A pseudonym or team name for a challenge entry
    """

    @classmethod
    def getURI(cls, id: Union[str, int]):
        return f"/evaluation/submission/{id}"

    def __init__(self, **kwargs):
        if not (
            "evaluationId" in kwargs
            and "entityId" in kwargs
            and "versionNumber" in kwargs
        ):
            raise KeyError

        super().__init__(kwargs)

    def postURI(self):
        return f"/evaluation/submission?etag={self.etag}"

    def putURI(self):
        return f"/evaluation/submission/{self.id}"

    def deleteURI(self):
        return f"/evaluation/submission/{self.id}"

SubmissionStatus ¶

Bases: DictObject

Builds an Synapse submission status object. https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatus.html

PARAMETER	DESCRIPTION
`id`	Unique immutable Synapse Id of the Submission TYPE: `Union[str, int]`
`status`	Status can be one of https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatusEnum.html.
`submissionAnnotations`	synapseclient.Annotations to store annotations of submission
`canCancel`	Can this submission be cancelled?
`cancelRequested`	Has user requested to cancel this submission?

Source code in synapseclient/evaluation.py

class SubmissionStatus(DictObject):
    """
    Builds an Synapse submission status object.
    <https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatus.html>

    Arguments:
        id: Unique immutable Synapse Id of the Submission
        status: Status can be one of
                   <https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatusEnum.html>.
        submissionAnnotations: synapseclient.Annotations to store annotations of submission
        canCancel: Can this submission be cancelled?
        cancelRequested: Has user requested to cancel this submission?
    """

    @classmethod
    def getURI(cls, id: Union[str, int]):
        return f"/evaluation/submission/{id}/status"

    def __init__(self, id: Union[str, int], etag: str, **kwargs):
        annotations = kwargs.pop("submissionAnnotations", {})
        # If it is synapse annotations, turn into a format
        # that can be worked with otherwise, create
        # synapseclient.Annotations
        submission_annotations = _convert_to_annotation_cls(
            id=id, etag=etag, values=annotations
        )
        # In Python 3, the super(SubmissionStatus, self) call is equivalent to the parameterless super()
        super().__init__(
            id=id,
            etag=etag,
            submissionAnnotations=submission_annotations,
            **kwargs,
        )

    # def postURI(self):
    #     return '/evaluation/submission/%s/status' % self.id

    def putURI(self):
        return f"/evaluation/submission/{self.id}/status"

    # def deleteURI(self):
    #     return '/evaluation/submission/%s/status' % self.id

    def json(self, ensure_ascii: bool = True):
        """Overloaded json function, turning submissionAnnotations into
        synapse style annotations.

        Arguments:
            ensure_ascii: (default = True) If false, then the return value can contain non-ASCII
            characters. Otherwise, all such characters are escaped in JSON strings.
        Returns:
            A Synapse-style JSON dictionary of annotations.
        """

        json_dict = self
        # If not synapse annotations, turn them into synapseclient.Annotations
        # must have id and etag to turn into synapse annotations
        if not is_synapse_annotations(self.submissionAnnotations):
            json_dict = self.copy()

            annotations = _convert_to_annotation_cls(
                id=self.id, etag=self.etag, values=self.submissionAnnotations
            )
            # Turn into synapse annotation
            json_dict["submissionAnnotations"] = to_synapse_annotations(annotations)
        return json.dumps(
            json_dict, sort_keys=True, indent=2, ensure_ascii=ensure_ascii
        )

Functions¶

json ¶

json(ensure_ascii: bool = True)

Overloaded json function, turning submissionAnnotations into synapse style annotations.

PARAMETER	DESCRIPTION
`ensure_ascii`	(default = True) If false, then the return value can contain non-ASCII TYPE: `bool` DEFAULT: `True`

Returns: A Synapse-style JSON dictionary of annotations.

Source code in synapseclient/evaluation.py

def json(self, ensure_ascii: bool = True):
    """Overloaded json function, turning submissionAnnotations into
    synapse style annotations.

    Arguments:
        ensure_ascii: (default = True) If false, then the return value can contain non-ASCII
        characters. Otherwise, all such characters are escaped in JSON strings.
    Returns:
        A Synapse-style JSON dictionary of annotations.
    """

    json_dict = self
    # If not synapse annotations, turn them into synapseclient.Annotations
    # must have id and etag to turn into synapse annotations
    if not is_synapse_annotations(self.submissionAnnotations):
        json_dict = self.copy()

        annotations = _convert_to_annotation_cls(
            id=self.id, etag=self.etag, values=self.submissionAnnotations
        )
        # Turn into synapse annotation
        json_dict["submissionAnnotations"] = to_synapse_annotations(annotations)
    return json.dumps(
        json_dict, sort_keys=True, indent=2, ensure_ascii=ensure_ascii
    )