Institutional Assessment with Plone
Nunez, Rafael
Rafael will describe the creation of the York College Assessment Management System (YAMS) plug-in for Plone. Using agile development and Plone's built-in functionality such as workflow, versions, roles to create a system that serves the College in having standardized but flexible, has continuity as leadership changes, and serves as a repository for accreditation.
 Good morning everyone. Welcome to day four of the talks for the Plone Conference. I'm your moderator for today, Chrissy Wainwright. Today I am introducing Rafael Nunez. He is a longtime Zope and Plone developer. He's been around the community since 2002, so quite a bit longer than me. Works for the City University of New York. Has done lots of things with flown and different add-ons. And today he is going to talk about institutional assessment with Plone. So Rafael, you can go ahead. Thank you. I'll share my screen. Okay, so institutional assessment with Plone. And I will explain the jam later. Okay, so Chrissy already introduced me. We run the York website, which is Power by Plone. We run a whole bunch of different systems within the college, like the parking permit system, faculty reappointment, they're all running on Plone. So in 2019, the college went for free accreditation and we were giving a warning. The warning was to improve our internal system of documenting assessment results, actions plan, and follow-up strategies. This is why the middle state commission of higher education, and we had a deadline of March 2020. The reason why this was happening is because there was no institutional assessment, there was no tool, college-wide for institutional assessment. Units were using Excel, Google Docs, Blackboard, or whatever tool they wanted to use at the time to collect and gather the information and fill out the documentation. The department shares will change over time within the accreditation period. So sometimes teams may get lost in the process when they're not passed properly from one share to the following. And the plans, even though they were available online, they were being distributed as PDF documents. So we had a web page which had all the different units as documents, plans, and reports, but there was no way to correlate the data. If you wanted to look at one particular part of the plan that correlates to different units, you couldn't do that because there's a whole bunch of documents all over the place. So we have eight months from concept to having a full year of data in this system. Eight months we have to develop the product, which didn't exist in our institution, as well as getting all the units. There were about 108 different units. There's 49 academics, 53 administrative units, and finances were an issue at York. Yeah, we were broke at the time due to contracts. The city of New York delayed for years and then they have to like, pretractively pay the whole bunch of salaries to everybody. And my team's way under staff. So the new director of institutional assessment called the meeting and asked if we can build it. She didn't consider getting another system because she has experienced with several in the past different institutions before she came to York. And they were very expensive for our budget and they didn't meet the needs. So they asked us, can we build it? And we say yes, we can. Or yes, Plon can. So we got the requirements. During the meeting, we got the requirements. The visitors should be able to look at the unit's mission and goal. The coordinator will be able to create the updated mission and goals, as well as plans, mid-year check and reports or annual year report. Then they will submit it to the director, which will review the information. And it will then be submitted to the VP for approval. They will look at the information, make any corrections and approve it. And then it will go to the committee. So there's a committee in the college, one for academic and one for non-academic units. And they will review the plans, mid-year checks and reports submitted. And they will make comments on it as well. And they will approve or project. So those were the basic requirements. So we created a plan. This was, I mean, we met during the summer. So I had to take time off because it's either user or lucid since I'm over my cabin in Anual Leaf. So by September 1st, we need to go live with the missions and goals. Phase two was going to be September 15, where we're going to have the non-academic unit plans. By the end of the month, we're going to have the academic unit plans ready to go to start populating. Phase four was the mid-year check. Phase five was the assessment reports. And this is now in January. And the reason why there's this gap is one because my team is on the staff and we have work to do in addition of creating this. But also all the units have to populate all this information. They need to gather data as well as creating the reports, mid-year checks and plans. And they have a due date when they have to do this task. So our phase was about 15 days before they have to start uploading and submitting information. That will give the units enough time to get trained on how to use the tool, how to upload your plans or mid-years or reports and be able to start uploading and submitting the information. And the last step is the institutional assessment department report. But they can see the big picture. And I will get to that in a second. So hammer time. It's time to start working on the tool, on the system. So we already have PLON. We are using PLON on everything that we develop within my team. It's usually PLON, not necessarily PLON. There's a tool for everything. But PLON fit most of the stuff that we do. We've used the distribution of PLON CASL for, we run with CASL for our main website. Because it had a lot of the things that we were going to do. It was the way we wanted to build PLON at a time we migrated in 2017. So instead of having to create all these different components, CASL had the distribution of things, I have envisioned for the PLON site at time. PLON has content rules. So that's something that we use for notifications every time they say transition. The notification will go out to the respective party. We needed to add the MEL2 role add-on for PLON. Because the default rules, you only have like reviewer, reader, and a couple of other roles that you can send to. With the MEL2 role, it allowed me to add new roles to the PLON site and being able to send notifications to those specific roles. LDAP authentication we use AD in the university. So our website is connected to that. All our faculty, staff, and students can log into the content management system using their AD credentials. Working copy and iterate is very important to me because we want to keep track of the changes over time. And pretty much you can play if something goes wrong. You know who to play. Workflow policy support. Our site is not only a public-facing site for our students but it also has a lot of sections that are intranet. So we use a workflow policy to set different policies in those areas. So if you were to log into a section of the website, that will be an intranet site. Also, all our faculty, staff, and students already have access to the CMS and a lot of the units are using it to update their own content, their websites. Faculty are using it for reappointment. We have another system which is a personnel and budget system running on the site which is how the faculty gets reappointed. And that's probably another talk after I upgrade to the Xperity or Vault tool. So in phase one, we created a unit's content type which is a container and goals. We created new roles, one for coordinator, department head, MVP. And the new roles, I in the past have been using Plum for a while. So in the past, I will just go to the CMI, add the new role, and I will have my role there to the CMI. This time, since I was creating that on, I also created roles to there because I want to have the auction from the sharing implant to be able to select those roles when given permission. This is restricted to site administrators, so only site administrator can assign those roles. And then we created the workflow. For the workflow, we created it through the web and then export it and put it into the product. We also created a view that will allow us to display the information the way they want. They have a specific vision of what the system should look like, how the documents should be presented. So we created a view for that. Also, this view will filter so the missions and goals when the documents are published are publicly available, but there's other portions of that template that are not displayed to the public. Okay, so our initial deployment. We have phase one ready. We have all the content created, and it's now time to deploy it. We plan a maintenance window. The website will not go down. We can keep this site running, but it will go into read only mode because we have to run build out our master server. So we have to announce it. So we announced the community that we'll try it will be in read only mode for like an hour. And before that, we have, I usually create a content on my machine locally, tested in the latest version of Plon. Currently, I'm probably in six of my box. So I'm sure that the product is up to date and working on all the versions. We deployed to our test development machine. We uploaded to our GitHub, our GitLab. And then we will upload tested on our clone of our production machine, our development, our castle development. Just to make sure, since I'm testing on the, I'm running the code in the latest version of my machine, but it's a Plon site is running castle, which is a different version. Just want to make sure that everything is, it's, it doesn't, doesn't break anything when you move it to castle. In some cases, I noticed that there's some, because of the different versions. So there's some things I have to change to make sure that there is working. So we test that at least a week before the go live date, the day before go live, what we do, everything is tested. We first upgrade our, one of our mirrors image of the master. Make sure that one will come back up when after running build out in non-news models. Just to make sure that nothing else breaks. And if that goes, I heard into plan, then I will upgrade mirrors two and one. So the day before the, the schedule maintenance, my mirrors are ready with a production code. All I have to do is bring down the master from build out and restart the clients. So, and that's exactly what we do. So after the initial build out, initial deployment of that on, we no longer need to run build out again. And we do this by just doing a, a kid pool on the different servers. Again, I will usually start with a mirror. If I can break a mirror, I have two other ones that can handle the information. So we will start by doing a kid pool, get the latest data tested on the, on one of the mirrors, make sure that doesn't break. Once that has been successfully run, they will run it on my other mirrors. And then I will do a kid pool on my master and sequentially restart the class of clients. So this website will never, never goes down, never goes into it only this way. So, and we're going to need this because on every face of this project, we're going to be uploading, updating the code and, and, and deploying it. Sometimes I will need to pre-install the product on Plum. I was doing this with a portal quick installer, which is going away or it's gone by now in Plum 6. Otherwise, I will have to go into the uninstall it and then pre-install it, which is pretty much the same thing that the portal quick installer, pre-install will do. Okay, so I guess it's demo time. Okay, so basically, and I'm putting this on my local machine here. The units are very simple. We have a vocabulary for the different units. So I'm just going to pick information technology. And then we have the division. This is part of administrative affairs. And then I will put the mission. So I'm just going to throw in some mission here. So that creates my unit. Like I said, again, we had 108 units at the beginning, which we had cut down last year because there were too many administrative units. So they were condensed administrative. So we, we have the mission and goal for this, this unit. And now we can add the goals. When we had this ready and deployed to the server, since we already had the missions and goals on the website, we started copying that information over. So we had a part-time worker just go through all the different units, create all of them, copy and paste all the missions and goals. I'm just going to add a new goal. Again, I'm just going to make something up here. First goal and then I'm going to add another goal. And you can add as many goals as you want. Now, you notice I didn't use the add button on the left menu. You can still add it through there. But the reason why we did it this way is to guide the users on what they need to do. In addition, we didn't create views for goals. We wanted to see the unit, not the individual goals, but they could if it needed. In addition, that allow us to present information in a more comprehensive way for them. After the mission and goal was populated, the units were published. And on their websites, instead of having data static missions and goals, we switched to a Ajax call to pull that information from the Jam system. Now, this information up here is publicly available for most units. Right now, this is private, so it's not going to be publicly available. But when it's published, it is publicly available. The table below, which I don't have any information yet, that's only visible to those with permission to add content or review the content. So in phase two, which was 15 days after the initial deployment, that will give us enough time for the units to be populated because we're going to need that first before anything else. So we went into phase two. In phase two, we created a the unit assessment plan, which is a container. And that container will also add a whole, the content, plan activity, programs, plan rationale, and collection evaluation. We created a new workflow for this content type, which is also followed by the containing objects in which it will follow what I described before. The coordinator can create the plan, then it will submit it to the director, and the other director will review it and submit it to the VP. And the VP will then approve it and it goes to the committee. And the committee will publish or publish either internally or externally. The workflow allows us to publish internally or externally. At the time, this information needed to be externally published, because of our accreditation. It's now most of the content is internally published. And they can switch back in for us, need it. And a new view was created for this program, for this unit assessment unit plan. So back to the demo. So I'm just going to create a new plan. We have a little instruction here, a little posted note, you can say to let the users know what to do. And then they can add a new plan. For the plan, all they have to do is pick the year and who's the unit director. So they pick the year and the unit director. That's all they choose. But that's not all the content that is contained by this content type. After you save it, you're going to see that you have a new year here. And this is organized by year. So we've got the year, the plan, the mid-year check, and the report. It will save you when the document is done. It will save you when the documentation draft after it is submitted, it will show the date I was submitted. Okay, so this is the new plan. So as you can see, all we selected was the year and the unit director. And this pre-populated a lot of the information. So we can see we already have the year, we created a title based on the information we have in the fields. We got the year, which was selected by the user, the unit, which we pick up from the parent. When we create a plan, our vocabulary will look at the parent's unit name and will bring that information into the plan. We do this. So we have every plan, every report has the information about the unit, the division in it. So we have the year, the division, the unit, the director, who created the plan, other contributors, and the date. The date is currently planned because it's still in private. Once submitted, it will have that date there. Now we have also the mission and the mission is currently taken from the unit mission. So they don't have to type in. The idea is the information is consistent. So the less you have people typing the information in, the more consistent it's going to be. And then we go into the assessment plan. We have a little yellow notes on the things that when they need to do something. Usually we sometimes will do the blue bar when they can edit a section here. So if I were to add a new outcome, this is a child object within this plan container. And as you can see, we have a vocabulary here, which will basically look at the unit. I have two units with the same information. It will take the unit's goals and that you can choose from. So the goals are consistent. So if you declare that your goal is this, you will be able to select that on the plan. This is my unit outcome. This is something better. It has to align. The expected outcome has to align with the strategic initiatives for the college. This is another vocabulary that we created. And we have also the institutional learning outcomes. So they have to select an institutional learning outcome. So it's a random. And what is activity? So you have to document this. What are the steps that are going to be done to get to this outcome? Well... Okay, how are we going to make measure success? Let's say number of downloads for now. Number of... This is... S-I... Number of... A little bit... Okay. Let's see... There's... And how are we going to collect this information? Is it direct or indirect? And what's our target? Increase... First, 5, 10%. Okay. So when you save it, you will still looking at the plan content type. And you will see your units outcome here. And the information you put in. The rationale for the plan, again, this is another content target will be added to this object. So when you click on that and you have a text box here. Otherwise we're going to run out of time. And you can add as many as you need. That's one of the reasons why we are creating this. This thing also have multiple paragraphs. And again, the data collection. Once they create something, that little label that says you need something here, disappears. So now this will go to the review process. So again, they will just transition it. Like they will normally submit to, in this case, because of my role, I can submit to the department head or chair, or submit to the VP or Dean. I'm using the slash here because I'm using the same workflow for academic and non-academic units. And they call the VP, it's not a VP on the academic side of being the department head. It's not a department head. It's a chair. But they pretty much have the same roles. So after this is completed, it will be submitted. And the workflow will change. If we go back to the unit, you can see that the information here changed to the time when it was submitted. So that is time. Then, let's go back to the presentation. Because we don't this part. Now we move into phase three. This was done about less than a month after the initial deployment of the add-on. And this had the academic side of it. The academic and non-academic are very similar. Similar enough that you can copy and paste most of the code, but different enough that you need two different content types. And they have different fields, different requirements, and different labeling for the information. In this case, the academic units, they have a different plan activity because they have different requirements. But the rationale, plan rationale in collection evaluation, those are the same. But it has a different view as well. And I'm not going to show this because if I do, I won't have enough time. But it's very similar to what I showed you before. Then we had the mid-year check. This is very simple. It's just one content type with some choices and some text area. So we created the content type and we created a view for it. I'm just going to show you real quick. So when you do admit-year check, as you can see here, you don't add it from the menu, though you could. But that's not the way we tell them to do it. They add it by clicking here. Then again, basically the year, the unit director, they can choose from the menu. You can search this about 10,000 users that they can search by. So, and basically this is just a checkbox of whether you're on track to meet your goals or your outcomes or not. So I'm just going to say yes, yes, yes, yes. For now, I'm going to put it here. You can comment on any of this. The word optional is there. It was requested by the users because they thought it was, even though it wasn't a require field, they thought it was require. So they were asked to add the word optional on the label. Oops. I can select this stuff. Again, using all the plum mechanism to create forms, so the sterility and to validate information. So that's our admit-year, which is very short and very simple and straightforward. Then we go into phase five. This was a little bit more, we have more time, which was good, because it was required a little bit more of fields, content types, and also we have to modify the workflow to accommodate this one. We also have a report view, which is a view for the parent content type, and we have a generate report. Basically, the report is not created manually. You generate a report for the year. When you do that, what it does is that it grabs the plan outcome from this year's plan and bring it into the report in addition of grabbing the unit name, the summary, the mission, all that from the parent. But it also looks at the year before. When you generate a report for the 2021 plan assessment year, you are taking information from the 2021 plan and you're taking information from the 2020 report. I will show you that in a second. I'm just going to generate a report for this year first. In 2019-2020, the report was a table because that's the way they were doing it and that's the way they wanted to present the information. When it is created, you can see that we brought in my goals from this year, since we don't have any data on changes implemented, that is blank. But if we create a plan for this year, it will bring the changes implemented from the assessment report from the year before. Again, this will give you a little edit action where you can modify the information. Some things are frozen, so they're only for displaying the information, their store, what the user cannot modify at this time because on your plan, you say this is what you were going to do. But there's a few things that you can do. So it will bring all the information. Now you will see what's the target met or not. What are the findings and progress and the use of results among other things that are here. I'm going to cancel this because I want to show you the one with the yams. I created this template before. I'm just going to run the report for the 2021-2022. You can see that you have the blue line for things that are good. You don't need to add anything else here. But also it shows you the orange line on things that need completion. It will tell you complete this section. So this section needs to be completed. It will also bring in the changes implemented. So these are things that we reported on the 2020-2021 assessment report. So if you look here, we got our outcome one test, test, test, test. And outcome two test, test, test, test as well. So when the report for the 2021-2022 is created, you can see that those are already put in. And now we have a little complete this section posted now here. So they know that they have more work to do. So when they do that, it will bring in all the information. As you can see here, this unit goal is no longer exist. It's not it, but it's not changed since it will be storing the data. They cannot change the unit goal now because that was the goal that was supported last year. They can add a few information here. So if the change was implemented, they can say that. What's the current status and whether they, they was a budgetary consideration. They're not using this for budget yet, but the plan is that the decisions about the budget are going to be highly influenced by your reports. So let's say this is spending approval. So now that section is completed and you can see how it's no longer has a little posted note in it. So the user knows that it is completed. This goal needs to be completed. So if I complete this section, target was met. I don't know the findings. So progress may I just gonna say progress. So we can use of results. So we can say that. I'm just gonna do test. I'm not out of time for typing stuff here. Test. Budget I consideration. Yes, question made it to the farming head. Unit communication. Yes, I communicated with everybody. I'm going to send an email and say that and now my unit are complete. So now I have a full report that I will submit to my director. They will review it. Wait, I'm missing this part. They will review it and they will pass it on to the next to the VP for approval. And so on. Also, they can upload attachments here. So if I have an attachment of rubric survey or other. That can be uploaded. And my Chrome is acting up this week. So I cannot even set files anymore. See if I can drop works. Yeah, that can work works locally. So now you can have that attachment there. And again, because this is a, the reports a container. You can add as many attachments as you want. There's also other accomplishments. So if you, if you did something incredible and you want to highlight it on your own. On your report that's relevant and you can add that information. Again, you can add as many as you want. Okay. In phase six, then we went into the, into the institutional reports. And basically this is we created a whole bunch of endpoints to connect the data to power BI. In the past we have created reports in plan using where you will show you nice charts. But because we are developing the institutional research solution assessment. They are already using power BI. It was easier for them to see the information in power BI. And they can slice and dice the data there. So reports were created for this. So basically we created a couple of power BI views for this we created the data to plan using the CSV format. So the information is pulled from one to power BI. And now the institutional assessment. They can say see there's 108 units throughout the years they can select the specific year if they want. So this is 2021-22. So the first year we had 102 plans in play submitted. You can see the distribution of plans reports. You can say you just want the administrative ones and that will narrow it down. So power BI. This is something that I probably will make a view in plan. So they can get even more of today information. This has to be refreshed. If I create this in plan, it will be real time data that we can just just pull in directly and present it. So from here they can see how many was this state of all the submissions. They can narrow it down to a specific one. They can also see the specific strategic goal. Let's say they want to see this goal how many people how many units in using that that can be done as well. So this is just show that. So this is one for an overview. This one for the different units as well. So rules and mosaic. We use rules as I said before the standard rules did not meet the need that I had for this product. So we added the add on that allow us to send emails to specific roles. And that's something that I didn't show. So let me just show that real quick. My sharing. The is coordinator director is the apartment head. Share is BP beans are some of the roles that we added with this add on and they're available through the web. So in order to to be able to notify people with those roles we have to add the add on. And that took care of that we send notifications to the coordinator with the notifications to the committee with the notifications to the BP with the notifications to the to the to. If anything is sent back at any point which can happen on the notifications are sent back to the coordinator or creator. We are using mosaic for the layout of yams or jams is time for your college assessment management system. I didn't come up with a name. I didn't agree with the name but they are using the name and the name stock. And that's why we have the jam at the beginning of the presentation. So the view when they go into jams, they can do it through the login into the website is in the dashboard is a link to jams, or they can just go to your technology to slash jams. And they will get this page which has some information about training. We have the AC, which is academic program assessment, and we have the ASS, which is a non academic program assessment which is administrative educational students boring. On that view as well you have the pending review items. We got a column for the pending unit, coordinate unit head. And then we have the pp pending committee. And this is all done with mosaic, because I, if I have to develop it will take me more time than just doing it through the web. And it's the same so if you use to use to use to use to reduce recycle. And then we had we created 25 dexterity types, content types, some are folders, some are not. We create the 22 views or a script to generate the content. We created 11 vocabularies are used to the side, a lot of things are clearing content. The goals vocabulary to make query for that particular unit goals. And the year for the year. I will talk about that in a second. We created workflows on that we needed for all this information. We created you need three new rules that I mentioned before, and power BI report. So, miscellaneous. I recently updated the year vocabulary to exclude items plans that are ready created or been years are ready created from the list, because the faculty that users on they were selecting the same year twice so that was creating two same year, but we only show one so they will start creating a little bit of confusion there so we'll just add it modify that so it was a query. Make sure that if that year exists for that unit for plan don't show that on that plan menu vocabulary of years. Table is outcome on the 2019 like I mentioned before. No, not that one. So the 2019 report, the information was presented as a table, which is, they were used to like this spreadsheet to have that information. But that doesn't work on the web because the web is a lot narrower than the table is. And it doesn't work for printing because you cannot print it. You need a really wide piece of paper. So the 2019 looks like it still use the table, though we can change it very easily. There's other fields that changes as well. So the 2019 year after that, the view is this, this structure all content here, because it's easier for printing. And it can, it does fit on the screen. So you can get the table here on the changes implemented because those have less columns in that fits on the screen. And when you print it. Okay. Cut you off. No, no, I'm good. I was done anyway. That was my last lie. Okay. Well, thank you for showing us how your organization uses flown I think it's it's always interesting to see how other developers solve problems. Thank you for watching. If you can join in jitsi. If you have any questions for Rafael go ahead and join him there posting the link. And thank you Rafael for speaking today. Thank you.
 Good morning everyone. Welcome to day four of the talks for the Plone Conference.  I'm your moderator for today, Chrissy Wainwright. Today I am introducing Rafael Nunez.  He is a longtime Zope and Plone developer. He's been around the community since 2002,  so quite a bit longer than me. Works for the City University of New York.  Has done lots of things with flown and different add-ons. And today he is going to talk about  institutional assessment with Plone. So Rafael, you can go ahead.  Thank you. I'll share my screen.  Okay, so institutional assessment with Plone. And I will explain the jam later.  Okay, so Chrissy already introduced me. We run the York website, which is Power by Plone.  We run a whole bunch of different systems within the college, like the parking permit system,  faculty reappointment, they're all running on Plone. So in 2019, the college went for  free accreditation and we were giving a warning. The warning was to improve our internal system  of documenting assessment results, actions plan, and follow-up strategies. This is why the middle  state commission of higher education, and we had a deadline of March 2020. The reason why this  was happening is because there was no institutional assessment, there was no tool, college-wide for  institutional assessment. Units were using Excel, Google Docs, Blackboard, or whatever tool they  wanted to use at the time to collect and gather the information and fill out the documentation.  The department shares will change over time within the accreditation period. So sometimes  teams may get lost in the process when they're not passed properly from one share to the  following. And the plans, even though they were available online, they were being distributed  as PDF documents. So we had a web page which had all the different units as documents,  plans, and reports, but there was no way to correlate the data. If you wanted to look at  one particular part of the plan that correlates to different units, you couldn't do that because  there's a whole bunch of documents all over the place. So we have eight months from concept to  having a full year of data in this system. Eight months we have to develop the product,  which didn't exist in our institution, as well as getting all the units. There were about  108 different units. There's 49 academics, 53 administrative units, and finances were an issue  at York. Yeah, we were broke at the time due to contracts. The city of New York delayed for years  and then they have to like, pretractively pay the whole bunch of salaries to everybody.  And my team's way under staff. So the new director of institutional assessment called the meeting  and asked if we can build it. She didn't consider getting another system because she  has experienced with several in the past different institutions before she came to York.  And they were very expensive for our budget and they didn't meet the needs. So they asked us,  can we build it? And we say yes, we can. Or yes, Plon can. So we got the requirements.  During the meeting, we got the requirements. The visitors should be able to look at the  unit's mission and goal. The coordinator will be able to create the updated mission and goals,  as well as plans, mid-year check and reports or annual year report. Then they will submit it to  the director, which will review the information. And it will then be submitted to the VP for approval.  They will look at the information, make any corrections and approve it. And then it will go  to the committee. So there's a committee in the college, one for academic and one for non-academic  units. And they will review the plans, mid-year checks and reports submitted. And they will make  comments on it as well. And they will approve or project. So those were the basic requirements.  So we created a plan. This was, I mean, we met during the summer. So I had to take time off because  it's either user or lucid since I'm over my cabin in Anual Leaf. So by September 1st, we need to go  live with the missions and goals. Phase two was going to be September 15, where we're going to have  the non-academic unit plans. By the end of the month, we're going to have the academic unit  plans ready to go to start populating. Phase four was the mid-year check. Phase five was the  assessment reports. And this is now in January. And the reason why there's this gap is one because  my team is on the staff and we have work to do in addition of creating this. But also all the  units have to populate all this information. They need to gather data as well as creating the reports,  mid-year checks and plans. And they have a due date when they have to do this task. So our phase was  about 15 days before they have to start uploading and submitting information. That will give the  units enough time to get trained on how to use the tool, how to upload your plans or mid-years  or reports and be able to start uploading and submitting the information. And the last step  is the institutional assessment department report. But they can see the big picture.  And I will get to that in a second. So hammer time. It's time to start working on the tool,  on the system. So we already have PLON. We are using PLON on everything that we develop within  my team. It's usually PLON, not necessarily PLON. There's a tool for everything. But PLON  fit most of the stuff that we do. We've used the distribution of PLON CASL for, we run with CASL  for our main website. Because it had a lot of the things that we were going to do. It was the way  we wanted to build PLON at a time we migrated in 2017. So instead of having to create all these  different components, CASL had the distribution of things, I have envisioned for the PLON site at  time. PLON has content rules. So that's something that we use for notifications every time they  say transition. The notification will go out to the respective party. We needed to add the  MEL2 role add-on for PLON. Because the default rules, you only have like reviewer, reader,  and a couple of other roles that you can send to. With the MEL2 role, it allowed me to add new  roles to the PLON site and being able to send notifications to those specific roles. LDAP  authentication we use AD in the university. So our website is connected to that. All our faculty,  staff, and students can log into the content management system using their AD credentials.  Working copy and iterate is very important to me because we want to keep track of the changes  over time. And pretty much you can play if something goes wrong. You know who to play.  Workflow policy support. Our site is not only a public-facing site for  our students but it also has a lot of sections that are intranet. So we use a workflow policy to set  different policies in those areas. So if you were to log into a section of the website,  that will be an intranet site. Also, all our faculty, staff, and students already have access to  the CMS and a lot of the units are using it to update their own content, their websites. Faculty  are using it for reappointment. We have another system which is a personnel and budget system  running on the site which is how the faculty gets reappointed. And that's probably another talk  after I upgrade to the Xperity or Vault tool. So in phase one, we created a unit's content type  which is a container and goals. We created new roles, one for coordinator, department head, MVP.  And the new roles, I in the past have been using Plum for a while. So in the past, I will just go  to the CMI, add the new role, and I will have my role there to the CMI. This time, since I was  creating that on, I also created roles to there because I want to have the auction  from the sharing implant to be able to select those roles when given permission.  This is restricted to site administrators, so only site administrator can assign those roles.  And then we created the workflow. For the workflow, we created it through the web and then  export it and put it into the product. We also created a view that will allow us to display  the information the way they want. They have a specific vision of what the system should look  like, how the documents should be presented. So we created a view for that. Also, this view will  filter so the missions and goals when the documents are published are publicly available,  but there's other portions of that template that are not displayed to the public.  Okay, so our initial deployment. We have phase one ready. We have all the content created,  and it's now time to deploy it. We plan a maintenance window. The website will not go down.  We can keep this site running, but it will go into read only mode because we have to run build  out our master server. So we have to announce it. So we announced the community that we'll try  it will be in read only mode for like an hour. And before that, we have, I usually create a content  on my machine locally, tested in the latest version of Plon. Currently, I'm probably in  six of my box. So I'm sure that the product is up to date and working on all the versions.  We deployed to our test development machine. We uploaded to our GitHub, our GitLab.  And then we will upload tested on our clone of our production machine, our development,  our castle development. Just to make sure, since I'm testing on the, I'm running the code in the  latest version of my machine, but it's a Plon site is running castle, which is a different version.  Just want to make sure that everything is, it's, it doesn't, doesn't break anything when you move  it to castle. In some cases, I noticed that there's some, because of the different versions. So  there's some things I have to change to make sure that there is working. So we test that  at least a week before the go live date, the day before go live, what we do, everything is tested.  We first upgrade our, one of our mirrors image of the master. Make sure that one will come back  up when after running build out in non-news models. Just to make sure that nothing else breaks.  And if that goes, I heard into plan, then I will upgrade mirrors two and one. So the day before the,  the schedule maintenance, my mirrors are ready with a production code. All I have to do is bring  down the master from build out and restart the clients. So, and that's exactly what we do.  So after the initial build out, initial deployment of that on, we no longer need to  run build out again. And we do this by just doing a, a kid pool on the different servers. Again,  I will usually start with a mirror. If I can break a mirror, I have two other ones that can handle  the information. So we will start by doing a kid pool, get the latest data tested on the,  on one of the mirrors, make sure that doesn't break. Once that has been successfully  run, they will run it on my other mirrors. And then I will do a kid pool on my master  and sequentially restart the class of clients. So this website will never, never goes down,  never goes into it only this way. So, and we're going to need this because on every face of this  project, we're going to be uploading, updating the code and, and, and deploying it. Sometimes I will  need to pre-install the product on Plum. I was doing this with a portal quick installer,  which is going away or it's gone by now in Plum 6. Otherwise, I will have to go into the  uninstall it and then pre-install it, which is pretty much the same thing that the portal  quick installer, pre-install will do. Okay, so I guess it's demo time. Okay, so basically,  and I'm putting this on my local machine here. The units are very simple. We have a vocabulary  for the different units. So I'm just going to pick information technology. And then we have the  division. This is part of administrative affairs. And then I will put the mission. So I'm just going  to throw in some mission here. So that creates my unit. Like I said, again, we had 108 units  at the beginning, which we had cut down last year because there were too many administrative  units. So they were condensed administrative. So we, we have the mission and goal for this,  this unit. And now we can add the goals. When we had this ready and deployed to the server,  since we already had the missions and goals on the website, we started copying that information  over. So we had a part-time worker just go through all the different units, create all of them,  copy and paste all the missions and goals. I'm just going to add a new goal. Again,  I'm just going to make something up here.  First goal and then I'm going to add another goal.  And you can add as many goals as you want. Now, you notice I didn't use the add button on the left  menu. You can still add it through there. But the reason why we did it this way is to guide  the users on what they need to do. In addition, we didn't create views for goals. We wanted to  see the unit, not the individual goals, but they could if it needed. In addition, that allow us  to present information in a more comprehensive way for them. After the mission and goal was  populated, the units were published. And on their websites, instead of having data static  missions and goals, we switched to a Ajax call to pull that information from the Jam system.  Now, this information up here is publicly available for most units. Right now, this is  private, so it's not going to be publicly available. But when it's published, it is publicly available.  The table below, which I don't have any information yet, that's only visible to those with permission  to add content or review the content. So in phase two, which was 15 days after the initial  deployment, that will give us enough time for the units to be populated because we're going to need  that first before anything else. So we went into phase two. In phase two, we created a  the unit assessment plan, which is a container. And that container will also add a whole,  the content, plan activity, programs, plan rationale, and collection evaluation.  We created a new workflow for this content type, which is also followed by the containing  objects in which it will follow what I described before. The coordinator can create the plan,  then it will submit it to the director, and the other director will review it and submit it to the  VP. And the VP will then approve it and it goes to the committee. And the committee will  publish or publish either internally or externally. The workflow allows us to publish  internally or externally. At the time, this information needed to be externally published,  because of our accreditation. It's now most of the content is internally published.  And they can switch back in for us, need it. And a new view was created for this program,  for this unit assessment unit plan. So back to the demo. So I'm just going to create a new plan.  We have a little instruction here, a little posted note, you can say to let the users know  what to do. And then they can add a new plan. For the plan, all they have to do is pick the year  and who's the unit director. So they pick the year and the unit director. That's all they choose.  But that's not all the content that is contained by this content type. After you save it, you're  going to see that you have a new year here. And this is organized by year. So we've got the year,  the plan, the mid-year check, and the report. It will save you when the document is done.  It will save you when the documentation draft after it is submitted, it will show the date I  was submitted. Okay, so this is the new plan. So as you can see, all we selected was the year  and the unit director. And this pre-populated a lot of the information. So we can see we already  have the year, we created a title based on the information we have in the fields. We got the  year, which was selected by the user, the unit, which we pick up from the parent. When we create  a plan, our vocabulary will look at the parent's unit name and will bring that information into  the plan. We do this. So we have every plan, every report has the information about the unit,  the division in it. So we have the year, the division, the unit, the director,  who created the plan, other contributors, and the date. The date is currently planned because  it's still in private. Once submitted, it will have that date there. Now we have also the mission  and the mission is currently taken from the unit mission. So they don't have to type in.  The idea is the information is consistent. So the less you have people typing the information in,  the more consistent it's going to be. And then we go into the assessment plan.  We have a little yellow notes on the things that when they need to do something.  Usually we sometimes will do the blue bar when they can edit a section here. So if I were to add a  new outcome, this is a child object within this plan container. And as you can see, we have a  vocabulary here, which will basically look at the unit. I have two units with the same information.  It will take the unit's goals and that you can choose from. So the goals are consistent. So if  you declare that your goal is this, you will be able to select that on the plan. This is my  unit outcome. This is something better. It has to align. The expected outcome has to align with  the strategic initiatives for the college. This is another vocabulary that we created.  And we have also the institutional learning outcomes. So they have to select an institutional  learning outcome. So it's a random. And what is activity? So you have to document this. What  are the steps that are going to be done to get to this outcome?  Well...  Okay, how are we going to make measure success?  Let's say number of downloads for now.  Number of...  This is...  S-I...  Number of...  A little bit...  Okay.  Let's see...  There's...  And how are we going to collect this information?  Is it direct or indirect?  And what's our target?  Increase...  First, 5, 10%.  Okay.  So when you save it, you will still looking at the plan content type.  And you will see your units outcome here.  And the information you put in.  The rationale for the plan, again, this is another content target will be added to this  object.  So when you click on that and you have a text box here.  Otherwise we're going to run out of time.  And you can add as many as you need.  That's one of the reasons why we are creating this.  This thing also have multiple paragraphs.  And again, the data collection.  Once they create something, that little label that says you need something here, disappears.  So now this will go to the review process.  So again, they will just transition it.  Like they will normally submit to, in this case, because of my role, I can submit to the  department head or chair, or submit to the VP or Dean.  I'm using the slash here because I'm using the same workflow for academic and non-academic  units.  And they call the VP, it's not a VP on the academic side of being the department head.  It's not a department head.  It's a chair.  But they pretty much have the same roles.  So after this is completed, it will be submitted.  And the workflow will change.  If we go back to the unit, you can see that the information here changed to the time when  it was submitted.  So that is time.  Then, let's go back to the presentation.  Because we don't this part.  Now we move into phase three.  This was done about less than a month after the initial deployment of the add-on.  And this had the academic side of it.  The academic and non-academic are very similar.  Similar enough that you can copy and paste most of the code, but different enough that  you need two different content types.  And they have different fields, different requirements, and different labeling for the  information.  In this case, the academic units, they have a different plan activity because they have  different requirements.  But the rationale, plan rationale in collection evaluation, those are the same.  But it has a different view as well.  And I'm not going to show this because if I do, I won't have enough time.  But it's very similar to what I showed you before.  Then we had the mid-year check.  This is very simple.  It's just one content type with some choices and some text area.  So we created the content type and we created a view for it.  I'm just going to show you real quick.  So when you do admit-year check, as you can see here, you don't add it from the menu,  though you could.  But that's not the way we tell them to do it.  They add it by clicking here.  Then again, basically the year, the unit director, they can choose from the menu.  You can search this about 10,000 users that they can search by.  So, and basically this is just a checkbox of whether you're on track to meet your goals  or your outcomes or not.  So I'm just going to say yes, yes, yes, yes.  For now, I'm going to put it here.  You can comment on any of this.  The word optional is there.  It was requested by the users because they thought it was, even though it wasn't a require field,  they thought it was require.  So they were asked to add the word optional on the label.  Oops.  I can select this stuff.  Again, using all the plum mechanism to create forms, so the sterility and to validate information.  So that's our admit-year, which is very short and very simple and straightforward.  Then we go into phase five.  This was a little bit more, we have more time, which was good, because it was required a little  bit more of fields, content types, and also we have to modify the workflow to accommodate  this one.  We also have a report view, which is a view for the parent content type, and we have a  generate report.  Basically, the report is not created manually.  You generate a report for the year.  When you do that, what it does is that it grabs the plan outcome from this year's plan and  bring it into the report in addition of grabbing the unit name, the summary, the mission, all  that from the parent.  But it also looks at the year before.  When you generate a report for the 2021 plan assessment year, you are taking information  from the 2021 plan and you're taking information from the 2020 report.  I will show you that in a second.  I'm just going to generate a report for this year first.  In 2019-2020, the report was a table because that's the way they were doing it and that's  the way they wanted to present the information.  When it is created, you can see that we brought in my goals from this year, since we don't have  any data on changes implemented, that is blank.  But if we create a plan for this year, it will bring the changes implemented from the  assessment report from the year before.  Again, this will give you a little edit action where you can modify the information.  Some things are frozen, so they're only for displaying the information, their store, what  the user cannot modify at this time because on your plan, you say this is what you were  going to do.  But there's a few things that you can do.  So it will bring all the information.  Now you will see what's the target met or not.  What are the findings and progress and the use of results among other things that are here.  I'm going to cancel this because I want to show you the one with the yams.  I created this template before.  I'm just going to run the report for the 2021-2022.  You can see that you have the blue line for things that are good.  You don't need to add anything else here.  But also it shows you the orange line on things that need completion.  It will tell you complete this section.  So this section needs to be completed.  It will also bring in the changes implemented.  So these are things that we reported on the 2020-2021 assessment report.  So if you look here, we got our outcome one test, test, test, test.  And outcome two test, test, test, test as well.  So when the report for the 2021-2022 is created, you can see that those are already put in.  And now we have a little complete this section posted now here.  So they know that they have more work to do.  So when they do that, it will bring in all the information.  As you can see here, this unit goal is no longer exist.  It's not it, but it's not changed since it will be storing the data.  They cannot change the unit goal now because that was the goal that was supported last year.  They can add a few information here.  So if the change was implemented, they can say that.  What's the current status and whether they, they was a budgetary consideration.  They're not using this for budget yet, but the plan is that the decisions about the budget are going to be highly influenced by your reports.  So let's say this is spending approval.  So now that section is completed and you can see how it's no longer has a little posted note in it.  So the user knows that it is completed.  This goal needs to be completed.  So if I complete this section, target was met.  I don't know the findings.  So progress may I just gonna say progress.  So we can use of results. So we can say that.  I'm just gonna do test.  I'm not out of time for typing stuff here.  Test.  Budget I consideration.  Yes, question made it to the farming head.  Unit communication. Yes, I communicated with everybody.  I'm going to send an email and say that and now my unit are complete.  So now I have a full report that I will submit to my director.  They will review it.  Wait, I'm missing this part.  They will review it and they will pass it on to the next to the VP for approval.  And so on. Also, they can upload attachments here. So if I have an attachment of rubric survey or other.  That can be uploaded.  And my Chrome is acting up this week.  So I cannot even set files anymore.  See if I can drop works. Yeah, that can work works locally.  So now you can have that attachment there. And again, because this is a, the reports a container.  You can add as many attachments as you want.  There's also other accomplishments. So if you, if you did something incredible and you want to highlight it on your own.  On your report that's relevant and you can add that information.  Again, you can add as many as you want.  Okay. In phase six, then we went into the, into the institutional reports.  And basically this is we created a whole bunch of endpoints to connect the data to power BI.  In the past we have created reports in plan using where you will show you nice charts.  But because we are developing the institutional research solution assessment.  They are already using power BI. It was easier for them to see the information in power BI.  And they can slice and dice the data there.  So reports were created for this.  So basically we created a couple of power BI views for this we created the data to plan using the CSV format.  So the information is pulled from one to power BI.  And now the institutional assessment.  They can say see there's 108 units throughout the years they can select the specific year if they want. So this is 2021-22.  So the first year we had 102 plans in play submitted.  You can see the distribution of plans reports. You can say you just want the administrative ones and that will narrow it down.  So power BI. This is something that I probably will make a view in plan.  So they can get even more of today information. This has to be refreshed.  If I create this in plan, it will be real time data that we can just just pull in directly and present it.  So from here they can see how many was this state of all the submissions.  They can narrow it down to a specific one.  They can also see the specific strategic goal.  Let's say they want to see this goal how many people how many units in using that that can be done as well.  So this is just show that. So this is one for an overview. This one for the different units as well.  So rules and mosaic.  We use rules as I said before the standard rules did not meet the need that I had for this product.  So we added the add on that allow us to send emails to specific roles.  And that's something that I didn't show. So let me just show that real quick.  My sharing.  The is coordinator director is the apartment head.  Share is BP beans are some of the roles that we added with this add on and they're available through the web.  So in order to to be able to notify people with those roles we have to add the add on.  And that took care of that we send notifications to the coordinator with the notifications to the committee with the notifications to the BP with the notifications to the to the  to.  If anything is sent back at any point which can happen on the notifications are sent back to the coordinator or creator.  We are using mosaic for the layout of yams or jams is time for your college assessment management system.  I didn't come up with a name. I didn't agree with the name but they are using the name and the name stock.  And that's why we have the jam at the beginning of the presentation.  So the view when they go into jams, they can do it through the login into the website is in the dashboard is a link to jams, or they can just go to your technology to slash jams.  And they will get this page which has some information about training.  We have the AC, which is academic program assessment, and we have the ASS, which is a non academic program assessment which is administrative educational students boring.  On that view as well you have the pending review items. We got a column for the pending unit, coordinate unit head.  And then we have the pp pending committee. And this is all done with mosaic, because I, if I have to develop it will take me more time than just doing it through the web.  And it's the same so if you use to use to use to use to reduce recycle.  And then we had we created 25 dexterity types, content types, some are folders, some are not.  We create the 22 views or a script to generate the content.  We created 11 vocabularies are used to the side, a lot of things are clearing content.  The goals vocabulary to make query for that particular unit goals.  And the year for the year. I will talk about that in a second.  We created workflows on that we needed for all this information.  We created you need three new rules that I mentioned before, and power BI report.  So, miscellaneous. I recently updated the year vocabulary to exclude items plans that are ready created or been years are ready created from the list, because the faculty that users on they were selecting the same year twice so that was creating two  same year, but we only show one so they will start creating a little bit of confusion there so we'll just add it modify that so it was a query.  Make sure that if that year exists for that unit for plan don't show that on that plan menu vocabulary of years.  Table is outcome on the 2019 like I mentioned before.  No, not that one.  So the 2019 report, the information was presented as a table, which is, they were used to like this spreadsheet to have that information.  But that doesn't work on the web because the web is a lot narrower than the table is.  And it doesn't work for printing because you cannot print it.  You need a really wide piece of paper. So the 2019 looks like it still use the table, though we can change it very easily.  There's other fields that changes as well.  So the 2019 year after that, the view is this, this structure all content here, because it's easier for printing.  And it can, it does fit on the screen.  So you can get the table here on the changes implemented because those have less columns in that fits on the screen. And when you print it.  Okay.  Cut you off.  No, no, I'm good. I was done anyway.  That was my last lie.  Okay.  Well, thank you for showing us how your organization uses flown I think it's it's always interesting to see how other developers solve problems.  Thank you for watching. If you can join in jitsi. If you have any questions for Rafael go ahead and join him there posting the link. And thank you Rafael for speaking today.  Thank you.
Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_0_2195.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_2196_15467.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_15468_26567.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_26568_26903.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_26904_29039.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_29040_34487.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_34488_35267.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_35268_35339.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_35340_39899.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_39900_40007.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_40008_40235.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_40236_40511.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_40512_45179.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_45180_46127.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_46128_46835.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_46836_49547.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_49548_49595.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_49596_49751.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_49752_50051.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_50052_50387.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_50388_50627.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_50628_51047.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_51048_51251.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_51252_51431.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_51432_51491.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_51492_51707.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_51708_52319.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_52320_52547.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_52548_52799.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_52800_52931.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_52932_53063.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_53064_53111.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_53112_53207.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_53208_54203.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_54204_54407.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_54408_54539.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_54540_55055.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_55056_55355.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_55356_55511.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_55512_55583.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_55584_55739.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_55740_55799.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_55800_56291.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_56292_56555.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_56556_56675.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_56676_56807.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_56808_56891.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_56892_57203.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_57204_57563.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_57564_57791.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_57792_58235.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_58236_61547.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_61548_61847.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_61848_63299.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_63300_68711.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_68712_68903.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_68904_69911.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_69912_70235.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_70236_70751.png Institutional_Assessment_with_Plone_-_Rafael_Nunez_fHDPpBr2dtQ_70752_71400.png
Good morning everyone. Welcome to day four of the talks for the Plone Conference. I'm your moderator for today, Chrissy Wainwright. Today I am introducing Rafael Nunez. He is a longtime Zope and Plone developer. He's been around the community since 2002, so quite a bit longer than me. Works for the City University of New York. Has done lots of things with flown and different add-ons. And today he is going to talk about institutional assessment with Plone. So Rafael, you can go ahead. Thank you. I'll share my screen. Okay, so institutional assessment with Plone. And I will explain the jam later. Okay, so Chrissy already introduced me. We run the York website, which is Power by Plone. We run a whole bunch of different systems within the college, like the parking permit system, faculty reappointment, they're all running on Plone. So in 2019, the college went for free accreditation and we were giving a warning. The warning was to improve our internal system of documenting assessment results, actions plan, and follow-up strategies. This is why the middle state commission of higher education, and we had a deadline of March 2020. The reason why this was happening is because there was no institutional assessment, there was no tool, college-wide for institutional assessment. Units were using Excel, Google Docs, Blackboard, or whatever tool they wanted to use at the time to collect and gather the information and fill out the documentation. The department shares will change over time within the accreditation period. So sometimes teams may get lost in the process when they're not passed properly from one share to the following. And the plans, even though they were available online, they were being distributed as PDF documents. So we had a web page which had all the different units as documents, plans, and reports, but there was no way to correlate the data. If you wanted to look at one particular part of the plan that correlates to different units, you couldn't do that because there's a whole bunch of documents all over the place. So we have eight months from concept to having a full year of data in this system. Eight months we have to develop the product, which didn't exist in our institution, as well as getting all the units. There were about 108 different units. There's 49 academics, 53 administrative units, and finances were an issue at York. Yeah, we were broke at the time due to contracts. The city of New York delayed for years and then they have to like, pretractively pay the whole bunch of salaries to everybody. And my team's way under staff. So the new director of institutional assessment called the meeting and asked if we can build it. She didn't consider getting another system because she has experienced with several in the past different institutions before she came to York. And they were very expensive for our budget and they didn't meet the needs. So they asked us, can we build it? And we say yes, we can. Or yes, Plon can. So we got the requirements. During the meeting, we got the requirements. The visitors should be able to look at the unit's mission and goal. The coordinator will be able to create the updated mission and goals, as well as plans, mid-year check and reports or annual year report. Then they will submit it to the director, which will review the information. And it will then be submitted to the VP for approval. They will look at the information, make any corrections and approve it. And then it will go to the committee. So there's a committee in the college, one for academic and one for non-academic units. And they will review the plans, mid-year checks and reports submitted. And they will make comments on it as well. And they will approve or project. So those were the basic requirements. So we created a plan. This was, I mean, we met during the summer. So I had to take time off because it's either user or lucid since I'm over my cabin in Anual Leaf. So by September 1st, we need to go live with the missions and goals. Phase two was going to be September 15, where we're going to have the non-academic unit plans. By the end of the month, we're going to have the academic unit plans ready to go to start populating. Phase four was the mid-year check. Phase five was the assessment reports. And this is now in January. And the reason why there's this gap is one because my team is on the staff and we have work to do in addition of creating this. But also all the units have to populate all this information. They need to gather data as well as creating the reports, mid-year checks and plans. And they have a due date when they have to do this task. So our phase was about 15 days before they have to start uploading and submitting information. That will give the units enough time to get trained on how to use the tool, how to upload your plans or mid-years or reports and be able to start uploading and submitting the information. And the last step is the institutional assessment department report. But they can see the big picture. And I will get to that in a second. So hammer time. It's time to start working on the tool, on the system. So we already have PLON. We are using PLON on everything that we develop within my team. It's usually PLON, not necessarily PLON. There's a tool for everything. But PLON fit most of the stuff that we do. We've used the distribution of PLON CASL for, we run with CASL for our main website. Because it had a lot of the things that we were going to do. It was the way we wanted to build PLON at a time we migrated in 2017. So instead of having to create all these different components, CASL had the distribution of things, I have envisioned for the PLON site at time. PLON has content rules. So that's something that we use for notifications every time they say transition. The notification will go out to the respective party. We needed to add the MEL2 role add-on for PLON. Because the default rules, you only have like reviewer, reader, and a couple of other roles that you can send to. With the MEL2 role, it allowed me to add new roles to the PLON site and being able to send notifications to those specific roles. LDAP authentication we use AD in the university. So our website is connected to that. All our faculty, staff, and students can log into the content management system using their AD credentials. Working copy and iterate is very important to me because we want to keep track of the changes over time. And pretty much you can play if something goes wrong. You know who to play. Workflow policy support. Our site is not only a public-facing site for our students but it also has a lot of sections that are intranet. So we use a workflow policy to set different policies in those areas. So if you were to log into a section of the website, that will be an intranet site. Also, all our faculty, staff, and students already have access to the CMS and a lot of the units are using it to update their own content, their websites. Faculty are using it for reappointment. We have another system which is a personnel and budget system running on the site which is how the faculty gets reappointed. And that's probably another talk after I upgrade to the Xperity or Vault tool. So in phase one, we created a unit's content type which is a container and goals. We created new roles, one for coordinator, department head, MVP. And the new roles, I in the past have been using Plum for a while. So in the past, I will just go to the CMI, add the new role, and I will have my role there to the CMI. This time, since I was creating that on, I also created roles to there because I want to have the auction from the sharing implant to be able to select those roles when given permission. This is restricted to site administrators, so only site administrator can assign those roles. And then we created the workflow. For the workflow, we created it through the web and then export it and put it into the product. We also created a view that will allow us to display the information the way they want. They have a specific vision of what the system should look like, how the documents should be presented. So we created a view for that. Also, this view will filter so the missions and goals when the documents are published are publicly available, but there's other portions of that template that are not displayed to the public. Okay, so our initial deployment. We have phase one ready. We have all the content created, and it's now time to deploy it. We plan a maintenance window. The website will not go down. We can keep this site running, but it will go into read only mode because we have to run build out our master server. So we have to announce it. So we announced the community that we'll try it will be in read only mode for like an hour. And before that, we have, I usually create a content on my machine locally, tested in the latest version of Plon. Currently, I'm probably in six of my box. So I'm sure that the product is up to date and working on all the versions. We deployed to our test development machine. We uploaded to our GitHub, our GitLab. And then we will upload tested on our clone of our production machine, our development, our castle development. Just to make sure, since I'm testing on the, I'm running the code in the latest version of my machine, but it's a Plon site is running castle, which is a different version. Just want to make sure that everything is, it's, it doesn't, doesn't break anything when you move it to castle. In some cases, I noticed that there's some, because of the different versions. So there's some things I have to change to make sure that there is working. So we test that at least a week before the go live date, the day before go live, what we do, everything is tested. We first upgrade our, one of our mirrors image of the master. Make sure that one will come back up when after running build out in non-news models. Just to make sure that nothing else breaks. And if that goes, I heard into plan, then I will upgrade mirrors two and one. So the day before the, the schedule maintenance, my mirrors are ready with a production code. All I have to do is bring down the master from build out and restart the clients. So, and that's exactly what we do. So after the initial build out, initial deployment of that on, we no longer need to run build out again. And we do this by just doing a, a kid pool on the different servers. Again, I will usually start with a mirror. If I can break a mirror, I have two other ones that can handle the information. So we will start by doing a kid pool, get the latest data tested on the, on one of the mirrors, make sure that doesn't break. Once that has been successfully run, they will run it on my other mirrors. And then I will do a kid pool on my master and sequentially restart the class of clients. So this website will never, never goes down, never goes into it only this way. So, and we're going to need this because on every face of this project, we're going to be uploading, updating the code and, and, and deploying it. Sometimes I will need to pre-install the product on Plum. I was doing this with a portal quick installer, which is going away or it's gone by now in Plum 6. Otherwise, I will have to go into the uninstall it and then pre-install it, which is pretty much the same thing that the portal quick installer, pre-install will do. Okay, so I guess it's demo time. Okay, so basically, and I'm putting this on my local machine here. The units are very simple. We have a vocabulary for the different units. So I'm just going to pick information technology. And then we have the division. This is part of administrative affairs. And then I will put the mission. So I'm just going to throw in some mission here. So that creates my unit. Like I said, again, we had 108 units at the beginning, which we had cut down last year because there were too many administrative units. So they were condensed administrative. So we, we have the mission and goal for this, this unit. And now we can add the goals. When we had this ready and deployed to the server, since we already had the missions and goals on the website, we started copying that information over. So we had a part-time worker just go through all the different units, create all of them, copy and paste all the missions and goals. I'm just going to add a new goal. Again, I'm just going to make something up here. First goal and then I'm going to add another goal. And you can add as many goals as you want. Now, you notice I didn't use the add button on the left menu. You can still add it through there. But the reason why we did it this way is to guide the users on what they need to do. In addition, we didn't create views for goals. We wanted to see the unit, not the individual goals, but they could if it needed. In addition, that allow us to present information in a more comprehensive way for them. After the mission and goal was populated, the units were published. And on their websites, instead of having data static missions and goals, we switched to a Ajax call to pull that information from the Jam system. Now, this information up here is publicly available for most units. Right now, this is private, so it's not going to be publicly available. But when it's published, it is publicly available. The table below, which I don't have any information yet, that's only visible to those with permission to add content or review the content. So in phase two, which was 15 days after the initial deployment, that will give us enough time for the units to be populated because we're going to need that first before anything else. So we went into phase two. In phase two, we created a the unit assessment plan, which is a container. And that container will also add a whole, the content, plan activity, programs, plan rationale, and collection evaluation. We created a new workflow for this content type, which is also followed by the containing objects in which it will follow what I described before. The coordinator can create the plan, then it will submit it to the director, and the other director will review it and submit it to the VP. And the VP will then approve it and it goes to the committee. And the committee will publish or publish either internally or externally. The workflow allows us to publish internally or externally. At the time, this information needed to be externally published, because of our accreditation. It's now most of the content is internally published. And they can switch back in for us, need it. And a new view was created for this program, for this unit assessment unit plan. So back to the demo. So I'm just going to create a new plan. We have a little instruction here, a little posted note, you can say to let the users know what to do. And then they can add a new plan. For the plan, all they have to do is pick the year and who's the unit director. So they pick the year and the unit director. That's all they choose. But that's not all the content that is contained by this content type. After you save it, you're going to see that you have a new year here. And this is organized by year. So we've got the year, the plan, the mid-year check, and the report. It will save you when the document is done. It will save you when the documentation draft after it is submitted, it will show the date I was submitted. Okay, so this is the new plan. So as you can see, all we selected was the year and the unit director. And this pre-populated a lot of the information. So we can see we already have the year, we created a title based on the information we have in the fields. We got the year, which was selected by the user, the unit, which we pick up from the parent. When we create a plan, our vocabulary will look at the parent's unit name and will bring that information into the plan. We do this. So we have every plan, every report has the information about the unit, the division in it. So we have the year, the division, the unit, the director, who created the plan, other contributors, and the date. The date is currently planned because it's still in private. Once submitted, it will have that date there. Now we have also the mission and the mission is currently taken from the unit mission. So they don't have to type in. The idea is the information is consistent. So the less you have people typing the information in, the more consistent it's going to be. And then we go into the assessment plan. We have a little yellow notes on the things that when they need to do something. Usually we sometimes will do the blue bar when they can edit a section here. So if I were to add a new outcome, this is a child object within this plan container. And as you can see, we have a vocabulary here, which will basically look at the unit. I have two units with the same information. It will take the unit's goals and that you can choose from. So the goals are consistent. So if you declare that your goal is this, you will be able to select that on the plan. This is my unit outcome. This is something better. It has to align. The expected outcome has to align with the strategic initiatives for the college. This is another vocabulary that we created. And we have also the institutional learning outcomes. So they have to select an institutional learning outcome. So it's a random. And what is activity? So you have to document this. What are the steps that are going to be done to get to this outcome? Well... Okay, how are we going to make measure success? Let's say number of downloads for now. Number of... This is... S-I... Number of... A little bit... Okay. Let's see... There's... And how are we going to collect this information? Is it direct or indirect? And what's our target? Increase... First, 5, 10%. Okay. So when you save it, you will still looking at the plan content type. And you will see your units outcome here. And the information you put in. The rationale for the plan, again, this is another content target will be added to this object. So when you click on that and you have a text box here. Otherwise we're going to run out of time. And you can add as many as you need. That's one of the reasons why we are creating this. This thing also have multiple paragraphs. And again, the data collection. Once they create something, that little label that says you need something here, disappears. So now this will go to the review process. So again, they will just transition it. Like they will normally submit to, in this case, because of my role, I can submit to the department head or chair, or submit to the VP or Dean. I'm using the slash here because I'm using the same workflow for academic and non-academic units. And they call the VP, it's not a VP on the academic side of being the department head. It's not a department head. It's a chair. But they pretty much have the same roles. So after this is completed, it will be submitted. And the workflow will change. If we go back to the unit, you can see that the information here changed to the time when it was submitted. So that is time. Then, let's go back to the presentation. Because we don't this part. Now we move into phase three. This was done about less than a month after the initial deployment of the add-on. And this had the academic side of it. The academic and non-academic are very similar. Similar enough that you can copy and paste most of the code, but different enough that you need two different content types. And they have different fields, different requirements, and different labeling for the information. In this case, the academic units, they have a different plan activity because they have different requirements. But the rationale, plan rationale in collection evaluation, those are the same. But it has a different view as well. And I'm not going to show this because if I do, I won't have enough time. But it's very similar to what I showed you before. Then we had the mid-year check. This is very simple. It's just one content type with some choices and some text area. So we created the content type and we created a view for it. I'm just going to show you real quick. So when you do admit-year check, as you can see here, you don't add it from the menu, though you could. But that's not the way we tell them to do it. They add it by clicking here. Then again, basically the year, the unit director, they can choose from the menu. You can search this about 10,000 users that they can search by. So, and basically this is just a checkbox of whether you're on track to meet your goals or your outcomes or not. So I'm just going to say yes, yes, yes, yes. For now, I'm going to put it here. You can comment on any of this. The word optional is there. It was requested by the users because they thought it was, even though it wasn't a require field, they thought it was require. So they were asked to add the word optional on the label. Oops. I can select this stuff. Again, using all the plum mechanism to create forms, so the sterility and to validate information. So that's our admit-year, which is very short and very simple and straightforward. Then we go into phase five. This was a little bit more, we have more time, which was good, because it was required a little bit more of fields, content types, and also we have to modify the workflow to accommodate this one. We also have a report view, which is a view for the parent content type, and we have a generate report. Basically, the report is not created manually. You generate a report for the year. When you do that, what it does is that it grabs the plan outcome from this year's plan and bring it into the report in addition of grabbing the unit name, the summary, the mission, all that from the parent. But it also looks at the year before. When you generate a report for the 2021 plan assessment year, you are taking information from the 2021 plan and you're taking information from the 2020 report. I will show you that in a second. I'm just going to generate a report for this year first. In 2019-2020, the report was a table because that's the way they were doing it and that's the way they wanted to present the information. When it is created, you can see that we brought in my goals from this year, since we don't have any data on changes implemented, that is blank. But if we create a plan for this year, it will bring the changes implemented from the assessment report from the year before. Again, this will give you a little edit action where you can modify the information. Some things are frozen, so they're only for displaying the information, their store, what the user cannot modify at this time because on your plan, you say this is what you were going to do. But there's a few things that you can do. So it will bring all the information. Now you will see what's the target met or not. What are the findings and progress and the use of results among other things that are here. I'm going to cancel this because I want to show you the one with the yams. I created this template before. I'm just going to run the report for the 2021-2022. You can see that you have the blue line for things that are good. You don't need to add anything else here. But also it shows you the orange line on things that need completion. It will tell you complete this section. So this section needs to be completed. It will also bring in the changes implemented. So these are things that we reported on the 2020-2021 assessment report. So if you look here, we got our outcome one test, test, test, test. And outcome two test, test, test, test as well. So when the report for the 2021-2022 is created, you can see that those are already put in. And now we have a little complete this section posted now here. So they know that they have more work to do. So when they do that, it will bring in all the information. As you can see here, this unit goal is no longer exist. It's not it, but it's not changed since it will be storing the data. They cannot change the unit goal now because that was the goal that was supported last year. They can add a few information here. So if the change was implemented, they can say that. What's the current status and whether they, they was a budgetary consideration. They're not using this for budget yet, but the plan is that the decisions about the budget are going to be highly influenced by your reports. So let's say this is spending approval. So now that section is completed and you can see how it's no longer has a little posted note in it. So the user knows that it is completed. This goal needs to be completed. So if I complete this section, target was met. I don't know the findings. So progress may I just gonna say progress. So we can use of results. So we can say that. I'm just gonna do test. I'm not out of time for typing stuff here. Test. Budget I consideration. Yes, question made it to the farming head. Unit communication. Yes, I communicated with everybody. I'm going to send an email and say that and now my unit are complete. So now I have a full report that I will submit to my director. They will review it. Wait, I'm missing this part. They will review it and they will pass it on to the next to the VP for approval. And so on. Also, they can upload attachments here. So if I have an attachment of rubric survey or other. That can be uploaded. And my Chrome is acting up this week. So I cannot even set files anymore. See if I can drop works. Yeah, that can work works locally. So now you can have that attachment there. And again, because this is a, the reports a container. You can add as many attachments as you want. There's also other accomplishments. So if you, if you did something incredible and you want to highlight it on your own. On your report that's relevant and you can add that information. Again, you can add as many as you want. Okay. In phase six, then we went into the, into the institutional reports. And basically this is we created a whole bunch of endpoints to connect the data to power BI. In the past we have created reports in plan using where you will show you nice charts. But because we are developing the institutional research solution assessment. They are already using power BI. It was easier for them to see the information in power BI. And they can slice and dice the data there. So reports were created for this. So basically we created a couple of power BI views for this we created the data to plan using the CSV format. So the information is pulled from one to power BI. And now the institutional assessment. They can say see there's 108 units throughout the years they can select the specific year if they want. So this is 2021-22. So the first year we had 102 plans in play submitted. You can see the distribution of plans reports. You can say you just want the administrative ones and that will narrow it down. So power BI. This is something that I probably will make a view in plan. So they can get even more of today information. This has to be refreshed. If I create this in plan, it will be real time data that we can just just pull in directly and present it. So from here they can see how many was this state of all the submissions. They can narrow it down to a specific one. They can also see the specific strategic goal. Let's say they want to see this goal how many people how many units in using that that can be done as well. So this is just show that. So this is one for an overview. This one for the different units as well. So rules and mosaic. We use rules as I said before the standard rules did not meet the need that I had for this product. So we added the add on that allow us to send emails to specific roles. And that's something that I didn't show. So let me just show that real quick. My sharing. The is coordinator director is the apartment head. Share is BP beans are some of the roles that we added with this add on and they're available through the web. So in order to to be able to notify people with those roles we have to add the add on. And that took care of that we send notifications to the coordinator with the notifications to the committee with the notifications to the BP with the notifications to the to the to. If anything is sent back at any point which can happen on the notifications are sent back to the coordinator or creator. We are using mosaic for the layout of yams or jams is time for your college assessment management system. I didn't come up with a name. I didn't agree with the name but they are using the name and the name stock. And that's why we have the jam at the beginning of the presentation. So the view when they go into jams, they can do it through the login into the website is in the dashboard is a link to jams, or they can just go to your technology to slash jams. And they will get this page which has some information about training. We have the AC, which is academic program assessment, and we have the ASS, which is a non academic program assessment which is administrative educational students boring. On that view as well you have the pending review items. We got a column for the pending unit, coordinate unit head. And then we have the pp pending committee. And this is all done with mosaic, because I, if I have to develop it will take me more time than just doing it through the web. And it's the same so if you use to use to use to use to reduce recycle. And then we had we created 25 dexterity types, content types, some are folders, some are not. We create the 22 views or a script to generate the content. We created 11 vocabularies are used to the side, a lot of things are clearing content. The goals vocabulary to make query for that particular unit goals. And the year for the year. I will talk about that in a second. We created workflows on that we needed for all this information. We created you need three new rules that I mentioned before, and power BI report. So, miscellaneous. I recently updated the year vocabulary to exclude items plans that are ready created or been years are ready created from the list, because the faculty that users on they were selecting the same year twice so that was creating two same year, but we only show one so they will start creating a little bit of confusion there so we'll just add it modify that so it was a query. Make sure that if that year exists for that unit for plan don't show that on that plan menu vocabulary of years. Table is outcome on the 2019 like I mentioned before. No, not that one. So the 2019 report, the information was presented as a table, which is, they were used to like this spreadsheet to have that information. But that doesn't work on the web because the web is a lot narrower than the table is. And it doesn't work for printing because you cannot print it. You need a really wide piece of paper. So the 2019 looks like it still use the table, though we can change it very easily. There's other fields that changes as well. So the 2019 year after that, the view is this, this structure all content here, because it's easier for printing. And it can, it does fit on the screen. So you can get the table here on the changes implemented because those have less columns in that fits on the screen. And when you print it. Okay. Cut you off. No, no, I'm good. I was done anyway. That was my last lie. Okay. Well, thank you for showing us how your organization uses flown I think it's it's always interesting to see how other developers solve problems. Thank you for watching. If you can join in jitsi. If you have any questions for Rafael go ahead and join him there posting the link. And thank you Rafael for speaking today. Thank you.
institutional assessment with Plone. So Rafael, Chrissy already introduced me. We run the York system because she has experienced with several migrated in 2017. So instead of having to create into a section of the website, that will be an that on, I also created roles to there because I the same workflow for academic and non-academic easily. There's other fields that changes as