Digital Transformation Building Blocks

This post describes technology building blocks to enable digital transformation at Utah Valley University, but the principles and concepts are universal. Utah Valley University is digitally transforming. The faculty and staff are excited about the possibilities and are envisioning a better future for our students. Students should expect our transformation to yield richer experiences, enhanced learning activities, increased skill acquisition, and a less burdensome path to completion and success. Our faculty and staff should experience less administrative burden allowing them to provide exceptional care, exceptional accountability, and exceptional results to our students and one another.

To facilitate digital transformation, the Office of Information Technology (OIT) and Academic and Student Digital Services (ASDS) must also transform. They must provide reliable and easy-to-use technology solutions that faculty and staff can use to enhance their interactions with others and improve the products and services they deliver. OIT and ASDS must adapt, modernize, provide existing products and services at a reduced cost, and provide new products and services with exceptional customer service.

In general, products and services should be available via self-service, 24 by 7, and ample support to make technology consumers successful, satisfied, and even delighted. The remainder of this post describes architectures, principles, and philosophies intended to make these necessary changes and the dream described above possible.

Application Programming Interfaces

Application programming interfaces (APIs) make it easier for others to interact with applications, create and use alternative user interfaces, and use the available services for alternative and even unexpected purposes. When acquiring or developing an application, it must have an API, and preferably a RESTful one. Data, services, and processes needed to build an application are used to create an API. The resulting API is then consumed to deliver a user experience. Mashup Corporation by Andy Mulholland et al. is a short fictional read that illustrates these concepts and the enabling power of APIs.

To make them most valuable, APIs must be exposed and consumed through API management tools. APIs should be more than simple JSON-based CRUD interfaces; APIs should expose appropriate business logic so that API consumers cannot violate required business processes. Enforcing this principle allows others to build delightful user experiences without institutional concern about policy, practice, or process compliance.

Domain-Driven Design

Everyone should read at least the first two chapters of the book Implementing Domain-Driven Design by Vaughn Vernon! Here is my super-short summary of domain-driven design (DDD) and those chapters — bring domain experts and developers together to create a ubiquitous language embedded in the application code itself. In addition, define or determine bounded contexts wherein this language is valid. This exercise helps software developers genuinely understand the business processes they are being asked to automate. It also helps the business participants understand the code being written and allows them to question decisions, test assumptions, and find bugs before deployment. This collaborative group of business leaders and developers is “the team”; success or failure is in their hands.

Microservices

Microservices are an architectural style that will be used at UVU to create larger systems. Systems built using microservices are loosely coupled; I would even go as far as saying they are highly decoupled, implement a single business capability, have well-defined interfaces, and communicate using only these interfaces. The size of a microservice is governed by the associated bounded context, go, and read the DDD book! At UVU, an essential part of a microservices’ interface is its ability to raise events. Go figure out why.

Event-Driven Architecture (EDA)

Systems that poll are inefficient! Build systems that raise events so other systems don’t have to waste time and resources. You can keep asking me if you must do this, but you can be assured that when I change my mind, I’ll let you know. If you didn’t find the humor in the last sentence, then reread the links.

Application Acquisition

When we purchase applications, we should give preference and a strong preference to those running in the cloud. When we build services or applications, they should use the most abstract service offerings that make sense. In other words, we should not instantiate servers and consume storage and then build queues, notification services, etc. We should instead use services such as queues, notification systems, serverless functions, etc.

DevOps

DevOps is a culture and practice that we hope will result in rapid development, testing, and software deployment. We also hope this increases accountability by allowing those who develop an application to be responsible for running and supporting it. Nothing motivates a developer to fix a bug more than to wake them to fix it repeatedly. Teams, DDD teams, are in charge and responsible for the functionality, performance, and reliability of “their” products.

If those in the hardware world think you’re off the hook, think again. Software is eating the world. The days of interacting with network switches, routers, firewalls, servers, storage appliances, AV equipment, etc., are over. Learn to program, learn to configure hardware devices using programs, remember to use DevOps principles to configure, test, and deploy hardware platforms as rapidly as “other” developers – that’s right, you just became developers!

Where to Compute

We built data centers and populated them with servers, storage systems, and network components in the past. As CPU performance increased, computers became more able to run multiple applications, but stability due to unintentional application interaction made this approach intolerable.

We found ourselves with many underutilized servers running single applications to maintain reliability. Along came server virtualization enabling us to instantiate multiple virtual servers on each physical server. Over the past several years, the number of physical servers has diminished considerably.

Well, we’re in the middle of another paradigm shift. We are continuing our journey that will result in our compute and storage being somewhere else. Acquired applications will also run in the “cloud.” in either case, they will not be housed here.

Networks

Unlike server and storage, I believe we will have a wired and wireless network on campus for the foreseeable future. However, the way we deploy, configure, and maintain these networks will change drastically. Remember, software is eating the world, and networking is not an exception to the rule. Network components will be physically installed in some generic way and then configured remotely via software. In a DevOps fashion, when a problem occurs, you’ll figure out what went wrong in the configuration script, you’ll repair the script, you’ll test the script, and you’ll redeploy.

Final Thoughts – For Now!

We have a great team! Let’s pursue all this FUN with great enthusiasm. Let us share our best thinking with others: share code on GitHub, answer questions on StackOverflow, blog about your experiences, publish papers, present at conferences, participate in panel discussions. In short, learn, teach one another, and teach the world!

Indie Educational Technology

Introduction

Last week I spent some time at Davidson College discussing Personal APIs and Indie Educational Technology with faculty, students, and staff from several other institutions of higher education and commercial entities, Known and Reclaim Hosting, that facilitate this work. This was a fantastic gathering of bright people and I can’t wait to be with them again. Thanks to Kristen Eshleman for getting us together, to Ben Werdmuller and Erin Richey for their instruction, to Audrey Watters for her insightful description of indie, and to Tim Owens, Jim Groom, Kin LanePhil Windley and Troy Martin for always making me think better. After returning to my day job I found myself asking the question, what is “indie educational technology”?

According to CNN “If it’s cool, creative and different, it’s indie” and the Urban Dictionary defines indie as, “an obscure form of rock [music] which you only learn about from someone slightly more hip than yourself.” I had to travel to North Carolina to have people, more hip than myself, educate me! I thank them and hope they continue to help me progress.

While I’m pretty sure the term “indie” in this context meant “independent of vendors and personal”, for my purposes I’m going to define indie educational technology as information technology that is cool, creative and different used to enhance the educational process.

Indie technology benefits both students and educational institutions. Students have a greater sense of ownership and motivation when they (a) control their personal information and (b) are able to interact with institutions with both institutionally provided applications and alternative systems (McCombs, 1997). Institutions benefit by having alternative application-hosting options and are not unnecessarily burdened by housing personal student information with its associated liability.

For example, let’s consider a traditional, non-indie, university registration system. One or more centralized systems contain university information about courses, classrooms, and instructors. In addition to this information, these systems contain personal student information that students are required to submit to participate in the registration process. To register for classes, students present user credentials to the university provided registration system, register for classes, and end their interaction. In this model the university retains personal student information, insists that students use the university provided system, and refuses to make alternative systems possible or feasible.

Consider an alternative, the indie approach. Like the non-indie system described above, one or more centralized systems contain university information about courses, classrooms, and instructors. However, personal information is not retained in this system. A student’s personal information is housed in a student-controlled system. Students authorize the university’s system to access necessary information. In this case students register for classes by presenting credentials to a registration application of their choice. This registration system requests and receives authorization from the university and the student to acquire university and personal student information, respectively. They register for classes and end their interaction. In this model the registration application disposes of personal and university information it was exposed to, while the university system retains the information necessary to indicate successful registration. In this scenario students may choose to use the university provided registration tool or alternative systems. In addition, the university does not house personal information or bear the associated liability. That’s cool, that’s indie!

So what makes this technology cooler, more creative, and different from what currently exists? Let me suggest that it is because indie technology will have several characteristics:

  • Personal API Enabled
  • Substitutable
  • Open Source
  • Modular
  • API-Based
  • Event-Driven

Let me elaborate on each.

Personal API Enabled

A personal API (PAPI) is an interface to personal information and resources. The resource owner protects these resources through explicit authorization. There are at least three key benefits of developing and using a PAPI:

  1. A PAPI changes the expectations of users. They develop a sense of ownership of their information and resources and begin to expect institutions to respect their rights and privacy. The use of a PAPI at an institution of higher education yields a perfect opportunity to educate students about these issues and help them understand what they should expect from other vendors and providers. They benefit from the ability to disassociate from institutions by simply revoking authorization to their data.
  2. A PAPI eliminates the need for a single university or vendor-provided application that all users must interact with. Users interact with their PAPI using applications of their choice. Institutional systems request permission to access personal information through the PAPI to perform needed functions. Institutions may provide applications for users, but the PAPI facilitates the creation of alternatives.
  3. A PAPI frees institutional technology modules from having to unnecessarily hold personal information. Institutional systems request needed personal information through the PAPI to perform their functions. Institutions should find this attractive because they will house less personal information, reducing their liability.

Note that giving people a personal API and letting them control their data, doesn’t mean that they get to control the university’s data. A PAPI lets people control the data that is theirs. For example, their phone number is their data. Their grades, on the other hand belong to the University. In addition, if students exercise their right to not authorize university access to needed personal information, the university is not obligated to fulfill the desired student request. University policy and process must still be followed.

The institutional complement to the PAPI is what we call the University API (UAPI). Through the UAPI an institution protects its resources through explicit authorization. In our example indie class registration system, the UAPI would make course, classroom, and instructor information available to the student chosen registration application. In addition, the UAPI would record the necessary registration outcomes.

Substitutability

Substitutability is the ability to use alternative systems or services to accomplish specific functions and move from one platform to another with ease and at little expense. This is applicable to both users and institutions.

Substitutability benefits users by allowing them to move their systems and services to alternative providers. They are also free to choose alternative applications to perform functions of interest. Institutions should facilitate both by pursuing strategies that allow authorized access to institutional information and consume necessary personal information through a PAPI respecting the user’s expectations of privacy.

Institutions benefit as well. Their systems can be operated on multiple platforms and through the use of technologies such as the UAPI and the PAPI, alternative systems can be used to accomplish institutional functions. If institutional systems only work properly when hosted at a single provider or moving them is onerous, institutions leave themselves vulnerable and open to the policies and practices of that provider. The inability to easily substitute one provider for another brings us back to our current state of affairs.

Open Source

Created systems and services should be freely available to others. First, this is what the cool kids do – indie. Second, by making them and API definitions freely available, others are more likely to adopt the technologies. Wide adoption results in many smart, hip people working on the same problems, resulting in better solutions. Licensing them appropriately protects our ability to use the things we develop.

Modularity

Modularity facilitates and drives an increased pace of innovation. Each module should deliver a small set of functions within a single bounded context as defined in the domain-driven design process. While these modules can be created using various techniques, at Brigham Young University (BYU) we will be defining them as microservices. These microservices will result in stand-alone modules that are easily understood by developers and will encourage extremely loose-coupling, facilitating a building block mentality to building systems. This approach will drive innovation in the core processes of BYU. 

API-Based

Each module will have an API that enables communication to and from the module. The API simplifies the use of the module and abstracts away the internal implementation. This abstraction permits changing the underlying implementation while protecting systems that rely on the module’s API. 

Event-Driven

While not strictly necessary, event-driven architectures are more efficient and absolutely cooler than polling-based systems. I think this alone makes event-driven, modular design a part of indie technology!

In a polling-based system you only become aware of changes when you ask if changes have been made. For example, in a registration system you determine how many students have registered for a particular class by asking (polling) the system. In an event-driven architecture, each time a student registers for a class an event reflecting this activity is posted to interested listeners. This results in more efficient communication and more timely responses to change. What could be more indie?

Now What

At Brigham Young University we intend on building many, if not all, of our core academic systems and services using modules with the above characteristics. The result will be a collection of modules that perform core functions of the institution, but are likely usable by others.

I hope that we can find ways of including others outside of the BYU community in the creation of our functional modules, systems, and services. Including others will make our work better, but more importantly will result in definitions and implementations that are more generic, enabling others to use them more easily. Each module, system, and service will have the characteristics outlined above making their use elsewhere practical and possible.

Finally, I hope we can all find a way to meet regularly to showcase our attempts, failures, and successes. We, at BYU, are open to conferences, workshops, or other venues where we can all continue this discussion.

myCoreDump

Introduction

I hope you enjoy this core dump! The thoughts are so interrelated and connected it is difficult to optimize the presentation so you may need to apply your own defragmentation to get it. In addition, the order is not intended to indicate priority, it is all freaking important!

University API (UAPI)

When acquiring or developing an application it must have an API, and preferably a RESTful one. If the function of the application is core to university business then it should be exposed through the UAPI. If it is not a core function of most, if not all, educational institutions, we should expose the API through our API management tools, but it shouldn’t be part of the UAPI.

Personal API (PAPI)

When we build a system that will store personal / individual information we should consider how we might leave the information in the hands or possession of the individual and access it for our use through their personal API. Since no one yet has a personal API, for the time being we must provide that as well. This will require you to stretch your imagination and creativity, but that’s good for you.

Domain-Driven Design

Everyone should read at least the first two chapters of the book Implementing Domain-Driven Design by Vaughn Vernon! The super short summary – bring domain experts and developers together to create a ubiquitous language that is embedded in the code itself. In addition, define or determine bounded contexts wherein this language is valid. Without this you won’t understand how we’re going to build solutions and you won’t have a clue what is in and what is not in a microservice. Read it!

Microservices

Microservices are an architectural style that will be used at BYU to create larger systems. Systems built using microservices are loosely coupled, I would even go as far as saying they are highly decoupled, they implement a single business capability, they have well defined interfaces, and communicate using only these interfaces. The size of a microservice is governed by the size of the associated bounded context, go and read the DDD book! At BYU an important part of a microservices’ interface is its ability to raise events. Go figure out why.

Event-Driven Architecture (EDA)

Systems that poll are inefficient! Build systems that raise events so other systems don’t have to waste time and resources. You can keep asking me if you have to do this, but you can be assured that when I change my mind I’ll let you know. If you didn’t find the humor in the last sentence then go read the links again.

Application Acquisition

When we purchase applications we should give preference, strong preference, to those that run in the cloud. In fact, before we choose an application that is not available as a service choose someone in your group you don’t love and care about to come get my approval.

When we build services or applications they will run at Amazon and use the most abstract service offerings that make sense. In other words, we should not instantiate EC2 servers and S3 storage and then build queues, notification services, etc., but instead should use services such as SQS, SNS, Lambda Functions, etc.

DevOps

DevOps is a culture and practice that we hope will result in the rapid development, testing, and deployment of software. We are measuring the number of deployments / week, failures / week, and time to recovery. We are promoting small changes, thorough automated testing, and deployment to production often. Your team (the DDD team) is in charge and responsible for the functionality, performance, and reliability of “your” product. 

If those in the hardware world think you’re off the hook, think again. Software is eating the world, software is eating your world. The days of interacting with network switches, routers, firewalls, etc. are over. Learn to program, learn to configure hardware devices using programs, learn to use DevOps to configure, test, and deploy hardware platforms as rapidly as “other” developers – that’s right, you just became developers!

Where to Compute

In the past we built data centers and populated them with servers, storage systems, and network components. As CPU performance increased computers became more able to run multiple applications, but stability due to unintentional application interaction made this approach intolerable.

We found ourselves with many underutilized servers running single applications to maintain reliability. Along came server virtualization enabling us to instantiate multiple virtual servers on each physical server. Over the past several years the number of physical servers has diminished considerably.

Well, it is time for another paradigm shift. We are now embarking on a journey that will result in our compute and storage being somewhere else. We will take advantage of Amazon to deliver what our applications and services need to run. Acquired applications will also run in the “cloud”. in either case they will not be housed here. Resources used previously to purchase servers and storage, and maintain them will be redirected to this new endeavor.

Networks

Unlike server and storage, I believe we will have a wired and wireless network on campus for the foreseeable future. However, the way we deploy, configure and maintain these networks will change drastically. Remember, software is eating the world and networking is not an exception to the rule. Network components will be physically installed in some generic way and then configured remotely via software.

In a DevOps fashion, when a problem occurs you figure out what went wrong in the configuration script, you repair the script, you test the script, and you redeploy. Remember, we’ll be watching how often you deploy, how many failures occur, and how long it takes to recover.

The days of hugging these devices are over. If you want one to hug, you can have one of the old ones and keep it in your office – disconnected from the network of course.

Domain of Ones Own (DoOO)

As we embark on this new path it is a great time for you to consider contributing to the content of the Internet. Let your light so shine by getting a domain of your own and sharing your goodness and skills with others. get one at domains.byu.edu. Here you can blog your greatest thoughts, post content that you syndicate to Facebook, Twitter or other services. Go learn, learning is fun!

We are offering this service to all students because we believe they should understand more about how the Internet works. We believe they have much to offer the world and they need to know they can share it with little help from service providers. What they build is transportable to other hosting services and is theirs! In the future a DoOO will enable an individual to have a portfolio and expose this and much more through their personal API (PAPI).

Final Thoughts – For Now!

We have a great team! Let’s pursue all of this FUN with the greatest enthusiasm and Heaven will shine down on us. Let us share our best thinking with others: share code on github, answer questions on stackoverflow, blog about your experiences, publish papers, present at conferences, participate on panels. In short, learn, teach one another, and teach the world!

css.php