Feedback via the DTE User Group Task Force

[RikkeVestesen]

Hi Costas

We have been contacted to provide feedback as part of the DTE User Group Task Force, so our team made a list of targeted input to support the planning of the next releases of the European Digital Testing Environment (DTE).

After sending the list, we thought it would be useful for you as well, so here it is :)

1. First, we would find it very useful to have a chat function where the tester and the reviewer — or possibly a review team responsible for the test — can communicate about the tests. There is a need for testers and reviewers to be able to communicate during a test process. Ideally, the chat could be linked to or reference individual test cases.
2. We would also like the possibility to attach files, such as a PDF, to each conformance statement. This could, for example, be used to attach use cases we have described for the test.
3. In addition, we need a more user-friendly way for testers to indicate exactly which test cases they would like to submit for review/audit. At the moment, we have to manually keep track of all individual test session IDs. It would be much more efficient if the tester could, for example, click a button to submit a session for review. Then, within ITB, we would be able to see a complete list of the sessions that the tester considers ready for review.
4. We would also like to provide feedback on the collaboration itself. The dialogue with the ITB team has been very positive, and we receive feedback on new features and the support we need quickly.

[costas80]

Thanks for sharing @RikkeVestesen! Very interesting indeed!

Adding my comments to what you mentioned:

1. First, we would find it very useful to have a chat function where the tester and the reviewer — or possibly a review team responsible for the test — can communicate about the tests. There is a need for testers and reviewers to be able to communicate during a test process. Ideally, the chat could be linked to or reference individual test cases.

This is exactly what we've been discussing in #141 and the feature will be available in the upcoming release! So it will be also available to the DTE team to use (if they choose to).

2. We would also like the possibility to attach files, such as a PDF, to each conformance statement. This could, for example, be used to attach use cases we have described for the test.

Do you mean as part of the definition of the conformance statement's specification or as something that would result during/after testing? If the former then this would be covered by #213 (where test suites are mentioned but this really relates to conformance statements). With a rich text section you could provide links to documents to download but if this is not sufficient we could also foresee attachments that appear for download.

3. In addition, we need a more user-friendly way for testers to indicate exactly which test cases they would like to submit for review/audit. At the moment, we have to manually keep track of all individual test session IDs. It would be much more efficient if the tester could, for example, click a button to submit a session for review. Then, within ITB, we would be able to see a complete list of the sessions that the tester considers ready for review.

The "normal" way we foresee this happening is by adding an admin interaction step in test cases. If a test session executes and hits such an interaction it will pause until being reviewed. Such test cases can be filtered for directly in the session dashboard (search for pending admin interactions), but also you can receive automated email reminders (if you have enabled SMTP integration and have chosen to enable such notifications). Moreover, once #141 is implemented we can also ensure that any time a test session is pending an admin interaction an automated notification message is sent to the community admin(s). I think all this would tie together very nicely (unless I'm missing something).

The only flow that is not covered would be cases where a test session completely finishes and then you expect to do a manual evaluation adding comments and possibly overriding the result (see #208). If this is the case however, it would be an approach I would not promote because overriding a result should be an exceptional action rather than part of a typical workflow.

4. We would also like to provide feedback on the collaboration itself. The dialogue with the ITB team has been very positive, and we receive feedback on new features and the support we need quickly.

Thanks for the shoutout! We also value enormously your excellent feedback, ideas and willingness to try out and brainstorm new features. It's fantastic input for us moving forward!

[RikkeVestesen]

Hi Costas

Thank you for the response, see my comments here:

Thanks for sharing @RikkeVestesen! Very interesting indeed!

Adding my comments to what you mentioned:

1. First, we would find it very useful to have a chat function where the tester and the reviewer — or possibly a review team responsible for the test — can communicate about the tests. There is a need for testers and reviewers to be able to communicate during a test process. Ideally, the chat could be linked to or reference individual test cases.

This is exactly what we've been discussing in #141 and the feature will be available in the upcoming release! So it will be also available to the DTE team to use (if they choose to).

We are looking forward to this!

2. We would also like the possibility to attach files, such as a PDF, to each conformance statement. This could, for example, be used to attach use cases we have described for the test.

Do you mean as part of the definition of the conformance statement's specification or as something that would result during/after testing? If the former then this would be covered by #213 (where test suites are mentioned but this really relates to conformance statements). With a rich text section you could provide links to documents to download but if this is not sufficient we could also foresee attachments that appear for download.

Yes indeed #213 would cover this quite nicely. Furthermore, having the ability to add attachments to the html also would enable us to us the ITB as the source of truth. This way we won’t accidently have dead or broken links a few years from now as it will always be hosted on the ITB.

3. In addition, we need a more user-friendly way for testers to indicate exactly which test cases they would like to submit for review/audit. At the moment, we have to manually keep track of all individual test session IDs. It would be much more efficient if the tester could, for example, click a button to submit a session for review. Then, within ITB, we would be able to see a complete list of the sessions that the tester considers ready for review.

The "normal" way we foresee this happening is by adding an admin interaction step in test cases. If a test session executes and hits such an interaction it will pause until being reviewed. Such test cases can be filtered for directly in the session dashboard (search for pending admin interactions), but also you can receive automated email reminders (if you have enabled SMTP integration and have chosen to enable such notifications). Moreover, once #141 is implemented we can also ensure that any time a test session is pending an admin interaction an automated notification message is sent to the community admin(s). I think all this would tie together very nicely (unless I'm missing something).

The only flow that is not covered would be cases where a test session completely finishes and then you expect to do a manual evaluation adding comments and possibly overriding the result (see #208). If this is the case however, it would be an approach I would not promote because overriding a result should be an exceptional action rather than part of a typical workflow.

Thank you for the explanation. I agree that admin interaction steps could support part of the review process, especially when a test case is designed to pause for an explicit administrative review.
However, our need is slightly different. In our certification process, testers often run the same test case several times before they are actually ready to submit anything to us for review or audit. In many cases, they will only want to submit the latest successful session, or the session they consider representative of their final result.
For that reason, we would prefer that the tester has to make an active choice to submit a specific test session for review, rather than the session automatically becoming pending for review whenever it reaches an admin interaction step or completes a test case.
The key point for us is that “ready for review” is not necessarily determined by the execution of the test case itself. It is determined by the tester deciding that this particular session is the one they want us to assess.
Ideally, the tester would therefore be able to mark or submit an individual session for review, for example by clicking a button such as “Submit for review”. As administrators, we would then be able to see a clear list of the sessions that testers have explicitly submitted for review/audit.
This would help avoid confusion around multiple repeated test runs and would reduce the need for us to manually track individual session IDs outside the ITB.

4. We would also like to provide feedback on the collaboration itself. The dialogue with the ITB team has been very positive, and we receive feedback on new features and the support we need quickly.

Thanks for the shoutout! We also value enormously your excellent feedback, ideas and willingness to try out and brainstorm new features. It's fantastic input for us moving forward!

[costas80]

Thanks for the follow-up @RikkeVestesen ! A couple additional reactions from my side:

Yes indeed #213 would cover this quite nicely. Furthermore, having the ability to add attachments to the html also would enable us to us the ITB as the source of truth. This way we won’t accidently have dead or broken links a few years from now as it will always be hosted on the ITB.

Sounds good. I've added in our backlog the possibility of adding one or more attachment files to a conformance statement. From the administrator's perspective these would be added at specification or actor level (depending what the conformance statement is made on), and will result in a set of attached files being displayed as downloadable links in the conformance statement. Given this, and also the feature discussed in #213 allowing you to add a custom preamble to a conformance statement, you should be able to cover all your needs.

To be able to track this better (for us and for you), would you mind creating a separate issue for this (e.g. "Feature: Support attachments to conformance statements") and referencing it here? No need for elaborate descriptions - you can just reference this issue.

Ideally, the tester would therefore be able to mark or submit an individual session for review, for example by clicking a button such as “Submit for review”. As administrators, we would then be able to see a clear list of the sessions that testers have explicitly submitted for review/audit.

I'll need to think about this more to see how it fits in the big picture (you are the first user to bring up a workflow like this). I'm thinking however if with the existing features we have delivered and the pending ones in the pipe, this could potentially already be achieved. Here's what I have in mind:

• Based on Support for test session comments and overriding a session's result #208 (already available) you now have comments that users can add to specific test sessions. You can set these at any time but, importantly also at the end of a test's execution, quite easily (i.e. you don't need to look up tests, copy info etc from other screens - it fits in the workflow).
• Based on Searching across all test cases and their session comment #232 (in the pipe) you will be able to search specifically for sessions with comments.
• Maybe your workflow could be covered by simply asking users to add a specific comment once they complete the test session they want to have reviewed? You will be able to filter these directly and also add your own assessment/comments/result within the same feature (it seems to fit nicely there).

What do you think?

[costas80]

Hi @RikkeVestesen.

Yes indeed #213 would cover this quite nicely. Furthermore, having the ability to add attachments to the html also would enable us to us the ITB as the source of truth. This way we won’t accidently have dead or broken links a few years from now as it will always be hosted on the ITB.

We just published our latest nightly build that includes custom documentation for conformance statements (see #213). Within this documentation you can also reference files defined as community resources (e.g. PDFs) that basically give you the attachments you were looking for. We chose to use community resources as opposed to having separate attachments as this gives you more flexibility:

• You can reference files for download that are either remotely hosted (e.g. link to a wiki attachment) or bundled within the ITB (defined as community resources).
• You can display attachments, along with any other information you want, in the way that suits you best. In Feature Request: Add Landing/Home Section for each testsuite #213 I added an example of a table that includes links, descriptions and version info.

Ideally, the tester would therefore be able to mark or submit an individual session for review, for example by clicking a button such as “Submit for review”. As administrators, we would then be able to see a clear list of the sessions that testers have explicitly submitted for review/audit.

As mentioned in my earlier comment you can already do this:

• Support for test session comments and overriding a session's result #208 added support for comments.
• Searching across all test cases and their session comment #232 (which was recently published) allows filtering test sessions based on comments (having any comment, or having comments with certain texts).

Assuming what I mention above covers you, the pending point on this issue would be #141 for the messaging/notification support between users and admins.