================== Handbook for cAIte ================== The following documents are targeted to a teacher, who wants to implement a new patient in cAIte and cooperates with the TIME https://www.medizin.uni-tuebingen.de/de/das-klinikum/einrichtungen/institute/time to implement those. The document should give you an overview on which steps are needed. As cAIte is under constant development some information might already be outdated. Therefore, don't rely to much on the statements here. New patient case ================ Create a new patient in cAIte with existing feedback pipelines and existing questionnaires. * total time frame: 2 weeks (6-16 working hours) * initial patient case creation: 1-2 hours * initial testing patient: 1-4 hours * adjust patient case information: 0.5 hours * testing adjusted patient case information: 1 - 5 hours * (optional) adjust cAIte specific scoring/feedback to case with cAIte team: 1-2 weeks * final adjustments: 0.5 hours * basic documentation: 1 hours Initial creation ---------------- Create the patient case with all the relevant information including the "Main Case Information", "Config Options", "Personal Data / Description", "Character Background", "Character Communication Style", general "Scoring / Feedback", "Medical History", and specific "Scoring / Feedback". Depending on the "Medical History", and the "Scoring / Feedback" additional information needs to be supplied. Testing ------- The testing of the case is critical! As we are using a relatively complex machinery in cAIte to achieve adherence to the patient case information, many interactions can occur that cannot be for seen and often can be mitigated, if discovered during testing. Please do at least the following after adjusting the patient case information: 1. Check that the page flow displays the **correct forms**, by starting the case and clicking through all the pages that are displayed. Have only a very brief chat with the patient. 2. If forms are not displayed correctly, are missing or to many forms are displayed. **Fix this first!** 3. After checking, that all forms are displayed correctly, run an **extensive interview** where you do the task requested from the students **as good as you can**. You should achieve **all points** in the feedback. The chat should at least last 20 minutes or more, to see if it is possible for cAIte to keep the relevant information active over the full course of the chat. **Write down where the feedback or the dialog falls short.** 4. Adjust the (technical) config of the patient case. If the scoring is off and cannot be fixed within the case description, get in contact with Tino (Konstantin Sering) to improve the scoring. After that redo 3. or proceed with 5. 5. Run an **extensive interview** where you pretend to be a bad student, who misunderstood parts of the task or where you pretend not having the correct information for the scoring. Adhere to the task, but try to score as badly as possible. **Inspect the feedback and the scores.** Does cAIte correctly captures the behaviour of the "bad student"? Write down notes. 6. Fill in missing information, sharpen the information for the feedback, ... potentially get in contact with Tino (Konstantin Sering). 7. If you run a **research project**, check that you can export all the relevant data from the interview. Is **all the data present**, that is needed to address the research question? 8. After everything seems to work, do **one final check** of the wording in the page flow. Are all instructions worded in plain language keeping points of confusion to a minimum? Are translations to English or German necessary? 9. Freeze the case and **pilot the case with a collegue**. Basic documentation ------------------- If the piloting is successful, make screenshots of all relevant parts of the case, export all the case information, **write** down a short methods section / **technical documentation** on how the case is created. Data collection, export, and analysis ------------------------------------- Next steps are to create links, which are used to run the interviews. If a research project is associated, export the data, do the analysis and write the paper. Costs ----- Depending on the amount of feedback and scoring the costs can vary drastically, but overall the costs per 20-minute chat can be estimated of from ???? Euro to ???? Euro. New questionnaire ================= To create a new questionnaire in cAIte help of the cAIte team is needed. The overall time line usually should be within 2-3 weeks. * total time frame: 3 weeks (3.5 - 7 working hours) * prepare case and default questionnaire: 1 hours * request new questionnaire: 1-4 hours * test questionnaire including data export: 1 hour * final adjustments: 0.5 hours .. note:: A questionnaire is for data collection only. It does not interact with the interview and the patient case information and is NOT the same as automated feedback. Prepare case and default questionnaire -------------------------------------- Prepare a case and a form flow where the new questionnaire should be added. Lookup, which questionnaires are close to the new one, if any exist. Request new questionnaire ------------------------- Send an e-mail to Tino (Konstantin Sering) to request the new questionnaire, add all questions and possible answers to it. State to which case it should be added and at which point in time in the page flow. If an existing questionnaire is closely related, point this out. State, which data needs to be collected with this questionnaire and what the rationale is behind it. Test questionnaire ------------------ After you receive the confirmation that the questionnaire is implemented, test the questionnaire. Especially, test that the data export works fine and contains all the data you need for the research question at hand. Do this at least one week before the data collection starts. Final adjustments ----------------- Add the questionnaire to all cases you want to test. Check that the questionnaire is correctly displayed. (Does it need translation to English / German?) New automatic feedback ====================== After chatting with a virtual patient cAIte gives automatic feedback. There are two types of feedback: Firstly, online or **live feedback**, which is used to give feedback while chatting. An example for live feedback is the use of different communication strategies like "open questions", "mirroring", etc. Secondly, **feedback** which is given **after the chat is completed** and uses the whole chat history as input. Examples for this feedback include checks, if all relevant questions are asked and relevant information is retrieved. Current feedback ---------------- * live feedback communication strategies * feedback on case / medical history * feedback on a depression scale * feedback on a SPIKES * feedback on case and medical history + questions specific to gynecology * feedback on case and medical history + questions specific to urology * some more specific ones Create new feedback ------------------- Creating a new automatic feedback pipeline belongs to the more sophisticated endeavours, where time estimates are hard to do. * total time frame: 6 weeks (10-60 working hours) * initial feedback creation: 3-10 hours * initial testing feedback: 1-4 hours * adjust feedback: 0.5 hours * testing adjusted feedback: 1 - 5 hours * piloting feedback with around 10 chats: 5-10 hours * annotating piloted feedback: 5-10 hours * feedback optimization: 5-10 hours * final adjustments: 0.5 hours * basic documentation: 1 hours TODO Data sharing ============ As cAIte is a learning and research tool and we are interested in gaining and publicating insights on useage and quality of cAIte as an AI-learning tool. For that we usually aggree on some data sharing guidelines when cAIte is used in a collaboration. This data sharing aggreement settles possible data use and co-authorships for different research questions, before the first data is collected. Our experience is that having a data sharing aggreement up front helps in setteling dispute later on.