Visual Studio Team Services

Now it's time to focus on another revolutionary online service - Visual Studio Team Services - that seamlessly enables continuous integration, continuous deployment, and continuous delivery. In fact, it would be more appropriate to call it a suite of services available under a single name. VSTS is a PaaS provided by Microsoft and hosted on cloud. The same service is available as (Team Foundation Server (TFS) on on-premises data centers. All examples used in this book use VSTS.

According to Microsoft, VSTS is a cloud-based collaboration platform that helps teams share code, track work, and shipping software. It was previously known as Visual Studio Online (VSO) and recently been renamed as VSTS. It is an enterprise software development tool and service that enables organizations to provide automation facilities to their end-to-end application lifecycle management process, from planning, to deployment, to getting real-time feedback from software systems users. This increases the maturity and capability of an organization to deliver high quality software systems to their customers again and again.

Successful software delivery involves bringing numerous processes and activities together efficiently. These include executing and implementing various agile processes, increasing collaboration among teams, seamless and automatic transition of artifacts from one phase of SDLC to another phase, and deployment to multiple environments. It is important to track and report on these activities, to measure and take action on these findings, and ultimately improve the delivery process. VSTS makes it simple and easy. It provides a whole suite of services that enable the following:

• Collaboration among every team member by providing a single interface for the entire application lifecycle management process
• Collaboration among development teams using source code management service
• Collaboration among test teams using test management service
• Automated validation of code and packaging through continuous integration using Build Management service
• Automated validation of application functionality and deployment, and configuration of multiple environments through continuous deployment and delivery using Release Management service
• Tracking and work item management using Work Management service

Figure 4 shows all the services available from the VSTS top navigation bar:

Figure 4: VSTS services

Source code management service

The source code management service, also known as Version control system, is one of the flagship services from VSTS. Source code version control helps store the code in a repository that can either be centralized or distributed. It also helps in versioning the code. Versioning helps maintain multiple copies of the same files, with a new copy created when the code gets updated. It helps with viewing the history of the code, comparing code between different versions, and retrieving previous versions.

We will need a VSTS account to be created before it can be used. We will look into the details of creating a VSTS account later in this chapter. Creating an account also creates a team project collection in which team projects can be created. The team project collection is a container that provides a security boundary and additional services such as agent pools. After an account is created, the next step is to create a team project. After a team project is created, the user is automatically redirected to the project dashboard. Each team project is based on a type of process. This process was previously known as Process Templates in VSO. The process determines how the requirements are broken down into features or user stories and tasks. It also provides a mechanism to manage them through work-item tracking. There are three types of processes available out-of-the-box in VSTS:

• Scrum: The Scrum process is for teams who follow the Scrum framework for their application development lifecycle.
• Agile: The Agile process is for teams using Agile methodology.
• CMMI: The Capability Maturity Model Integration (CMMI) process is a comparatively more formal process for executing projects. It mainly focuses on continuous improvement through telemetry.

Each of the processes, whether Scrum, Agile, or CMMI, involves different ways of executing projects and demands a complete book for itself. We will not go into detail in this book.

The code link on the top navigation bar will take us to the source code management control panel. This is shown in Figure 5:

Figure 5: Source code link

A team project in VSTS is a security boundary and logical container that provides all the services we mentioned in the previous section. VSTS allows for the creation of multiple projects within a single account. By default, a repository is created with the creation of a project. However, VSTS allows for the creation of additional repositories within a single project. The relationship between the VSTS account, projects, and repository is shown in Figure 6:

Figure 6: Relationship between VSTS account, team projects, and repositories

VSTS provides two types of repositories, as follows:

• Git
• Team Foundation Version Control (TFVC)

It also provides the flexibility to choose between Git or TFVC source control repositories. There can also be combination of Team Foundation Server (TFS) and TFVC repositories available within a single project.

Team Foundation Version Control

TFVC is the traditional and centralized way of implementing version control in which there is a central repository, which developers work on directly in connected mode to check-in their changes. If the central repository is offline or unavailable, developers cannot check-in their code and have to wait for it to go online and become available. Other developers can see only the checked-in code. Developers can group multiple changes into a single change set for checking-in code changes that are logically grouped to form a single change. TFVC locks the code files that are undergoing edits. Other developers can read the locked file but they cannot edit it. They have to wait for the prior edit to be complete and release the lock before they can make their own edits. A history of check-ins and changes is maintained on the central repository, while the developers have a working copy of the files but not the history.

TFVC works very well with large teams that are working on the same projects. This allows control over source code at a central location. It also works the best when the project is for a long duration since the history is managed at a central location. TFVC has no issues working with large and binary files.

Exploring Git

Git, on the other hand, is a modern distributed way of implementing version control where the developers can work on their own local copies of code and history in offline mode. Each developer has a local copy of the code and its entire history. They can make changes to the code and can commit to the local repository. They can then connect to the central repository in order to synchronize their local repository on an as-needed basis. This allows every developer to work on any file since they will be working on their local copy only. Branching in Git does not create another copy of the original code and is extremely quick to create.

Git works well with smaller teams. With larger teams, there is a substantial overhead to manage multiple pull requests to merge the code on a central repository. It also works best for smaller duration projects as the history will not get too large to be downloaded and manageable on every developer's local repository. Branching and merging is a breeze with advanced options.

Git is the recommended way of using source control because of the rich functionality it provides. We will use Git as repository for our sample application in this book.

Build Management service

Another very important service in VSTS is the Build Management service. The main task of the Build Management service is to provide continuous integration service to projects.

As we already know by now, continuous integration is the process of compiling, building, and validating code. It is about deploying code onto a test environment and validating the code quality, code coverage, and whether the code bits are in working condition. Build services help automate this entire process. Similar to source code management, the build service is scoped at the team project level. In each team project, VSTS allows the creation of multiple build definitions and templates. Each build definition is attached to a branch of a repository. This could be a Git or a TFVC repository. Templates act as the basic building blocks for definitions. They can be a starting place for the creation of custom build definitions, with activities already defined in the definition. A build definition can also be saved as a template and reused to create other build definitions. In this section, we will introduce the concepts first and then show how it is implemented or configured.

The build definition consists of multiple activities that run in sequence, one after another, like a pipeline. Each activity is responsible for executing a single task within the overall build pipeline. Figure 7 shows an example of a build definition consisting of six tasks created using a Visual Studio build template:

Figure 7: Sample build definition based on a Visual Studio template

These tasks are responsible for the following:

• Restoring the NuGet packages needed by the solution
• Building the entire solution, which in turn builds every project within it
• Executing unit tests on the compiled code
• Publishing a symbol path that helps in debugging
• Copying the generated and compiled assemblies to a destination folder
• Publishing the artifacts on the VSTS server

There are many more tasks available which can be used to augment the build definition further. Figure 8 shows the tasks available for build definition. There is also a marketplace where more tasks can be made available to the build definition. It is important to note at this stage that the same tasks are available for both build and release definitions. There is no difference in their configuration and execution, whether they are executed as part of a build or release definition:

Figure 8: Build tasks

Executing Build Definitions

Build definitions need build environments in order to be executed. Tasks in a build definition have pre-requisites which should be available on build servers. VSTS provides built-in build servers out-of-the-box with prerequisites already installed on them. They are available to every account for the execution of build definitions. However, VSTS is flexible enough to accept our own custom-built servers for the execution of build definitions.

Build architecture

Figure 9 depicts the VSTS build architecture. Every build server should have build agents installed and running on it. Build agents are Windows services configured to interact and work with the VSTS build service. The agents are grouped together to form an agent pool. An agent pool is defined at the VSTS account level and there can be multiple agent pools defined. VSTS provides a default agent pool, named Hosted. All VSTS-provided servers are part of the hosted agent pool:

Figure 9: Relationship between agents, pools, and queues

Agents, agent pools, and agent queues

The relationship between build servers, agents, agent pools, and agent queues is shown in Figure 9.

Multiples can be installed on a build server.

Each of these agents belongs to a single agent pool. An agent cannot belong to multiple agent pools.

An agent queue is mapped to a single agent pool. A queue cannot be mapped to multiple agent pools however a single agent pool can be used by multiple agent queues.

Queuing Build Definitions

Build definitions are queued before they can be executed. When queuing a build definition, the name of the agent queue on which it should be queued should be provided. A queue is tied and mapped to an agent pool. Any available agent in the pool that meets the capabilities needed by the build definition eventually picks up the request and executes it.

Configuring a Build definition

The Build Management control panel can be reached via the Builds sub-menu in the Build & Release link on the top navigation bar. Clicking on the + New button available at the top-right of the window will start a new build definition wizard. Figure 10 shows the Create new build definition wizard:

Figure 10: Creating a build definition

Selecting a Build template and providing an appropriate repository name, branch name, and agent queue information will create a build definition in draft mode with a few activities in it. The build definition control panel is shown in Figure 11:

Figure 11: Build control Panel

There are two levels of configuration for each build definition - Build level and Task level configuration.

• Build level configuration: These configurations affect the entire build definition and execution process. It also provides the options to save and queue the build definition from any tab:
  • Build: This tab lists every task that comprises the build definition. Configuration for every task can be done from this tab. Refer to Figure 11 for a visual example.
  • Options: This tab is used to build multiple configurations such as debug, release, or any other configuration defined in the project. Selecting Multi-Configuration shows further options. We can provide comma-separated configuration names, select whether each of these configurations should run in parallel or in sequence, and select whether these configurations should stop if an error occurs. It also allows us to create a new work item (Bug, Epic, Feature, Task, Test case, User story, Issue) when the build fails by checking the Create Work Item on Failure checkbox. This is shown in Figure 12:
    

    Figure 12: Options tab for build definition

  • Repository: The options shown here are different based on the type of repository. For a Git-based repository type, information about the repository and its branch in the project should be provided. The Clean option removes untracked files and branches from the repository. Label sources helps provide labeling for the source code files and versions them for easy identification. To report on the build status the corresponding option should be checked. If code is reused from another repository, the sub module should also be checked out. This is shown in Figure 13:
    

    Figure 13: Repository tab of build definition

  • Variables: This tab helps make the build definition generic and reusable. It helps remove hard coding and changes the behavior of the build definition during execution. It also helps use the same value in multiple places within the same definition. There are a set of predefined variables available out of box. In addition, new variables can be defined by the user. Each variable has a name and a value. A variables can also be encrypted using the secret lock button. Moreover, Allow at Queue Time enables the availability of the same variable during manual queuing providing an opportunity to change its value before execution. Figure 14 shows the Variables tab in a build definition:
    

    Figure 14: Variables tab of a build definition

  • Triggers: A build can be initiated manually as a scheduled activity and/or as a continuous integration. When Continuous integration (CI) is chosen, any commit or check-in of code will initiate the build process by queuing it to an agent queue. Figure 15 shows the Triggers tab of the build definition:
    

    Figure 15: Triggers tab of the build definition

  • General: This tab allows us to choose the queue for the submission of our build definition. If the build needs access across projects in the account, Project Collection should be chosen as the authorization scope. Otherwise, Current Project should be chosen. Description provides more information about the build definition. Each build is given a unique number and if we want more useful names for the build, Build number format can be used to define them. The Demands section lists the capabilities a build agent must possess for successful execution of the definition. A build will not execute if the demands are not satisfied. Build job timeout in minutes provides control to VSTS to cancel the build execution if it does not finish within the given time. Figure 16 shows the General tab of a build definition:
    

    Figure 16: General tab of a build definition

  • Retention: VSTS stores logs and other information for every build execution. This tab allows setting the number of days to retain various aspects of build output. Figure 17 shows the Retention tab:
    

    Figure 17: Retention tab of the build definition

  • History: VSTS maintains the history of changes made to the build definition. This was not possible in earlier versions of VSTS. It provides version control for the build definition itself and allows reversion back to a previous version and comparison between multiple versions of the same definition:
    

    Figure 18: History tab of the build definition

    There are further configuration options available through the Build Definition context menu. Among the most important of them is the Security configuration. The Security configuration allows us to provide contributor, administrator, and authoring permissions to users and groups. This is shown in Figure 19:

    

    Figure 19: Build definition security configuration

    The Queue build button lists a build definition to a queue and executes it.

• Task level configuration: Each task has its own set of configurations to work on. There are different requirements for different tasks. Figure 20 shows the Build solution task. This is responsible for compiling every project within a solution. It accepts the solution file path and multiple configurations related to NuGet. These include path to NuGet.config, whether to restore or install NuGet packages, disabling the NuGet cache, and NuGet arguments. The advanced section accepts the path to NuGet.exe:
  

  Figure 20: Build solution task within the build definition

• The Control Options section has an additional configuration that controls the entire task. These configurations are common to all the tasks available in VSTS build and release definitions. Control Options is shown in Figure 20.
• The Enabled check refers to whether this task participates in build execution. It is not executed if the check is removed. By default, it is in the checked state.
• A build execution fails when any task within it fails. The Continue on error check helps change this setting and allows us to continue executing the build even if this tasks fails.
• Always run ensures that a given task always executes, even if there are build failures from other tasks.

Release Management service

Now it's time to look at how VSTS can help with continuous deployment and continuous delivery. We already know by now that continuous deployment refers to the deployment of application on multiple environments, including pre-production and production environments, through automation. This involves both the provisioning and configuration of environments and applications.

The Release Management service helps automate the deployment and configuration to multiple environments. It helps execute validation tests, such as functional tests, on these environments. The Release Management service help implement both continuous deployment and delivery. Similar to Source Code and Build mMnagement, the Release Management service is scoped at the project level. For each project, VSTS allows the creation of multiple release definitions.

Each release definition consists of multiple environment definitions, each environment representing a deployment target. Test, staging, and production are examples of environments. Release Management executes these environment definitions and each environment can be configured to run in parallel or in sequence after a prior environment execution. Each environment definition would typically consist of provisioning and configuring multiple servers, deploying and configuring application components on these servers, and validating the application. Although the configuration of environments within a release definition could be different, from a DevOps perspective, they should be similar.

Each environment definition consists of multiple activities that run in sequence, one after another, like a pipeline. Each activity is responsible for executing a single task within the overall environment pipeline. Figure 21 shows a release definition example consisting of two environments, authoring and production, each with multiple activities in its pipeline. Each contains the same tasks. However, their configuration is different for each environment:

Figure 21: Release definition with two environments

It may be recollected that tasks are available to both build and release definitions are common in VSTS. Figure 8 in this chapter shows the rich set of tasks available for build and release definitions.

The release pipeline gets its inputs in the form of artifacts. One of the final steps in continuous integration is to drop the generated package to a specified location. Release Management considers the output from a build execution as an artifact. It also considers files stored in Git and TFVC repositories as artifacts.

Executing release definitions

Release definitions need release environments in order to be executed. The tasks in a release and environment definition have pre-requisites, and those should be available on the release servers. VSTS provides built-in release servers out-of-the-box, with prerequisites installed on them. They are available to every account in VSTS for the purpose of executing release definitions. However, at the same time, VSTS is flexible enough to accept our own custom release servers for the execution of release definitions.

Release Management architecture

The Release Management architecture is the same as that of Build Management, which is depicted in Figure 9. Every release server should have a release agent installed and running on it. Release agents are Windows services that are configured to interact and work with the VSTS Release Management service.

Executing a release definition in turn executes the environment definitions. Each release environment is configured to use an agent queue. This agent queue selected during release definition creation becomes the agent queue configuration for the first environment. Each environment definition can be configured to use a particular agent queue and a request is queued on the agent queue for each environment. An agent queue is formed at the project-collection level and multiple queues can be created. When queuing an environment definition, VSTS expects the artifact that should be used within its pipeline.

Release definition configuration

The Release Management control panel can be reached through the Release sub-menu of the Build & Release link on the top navigation bar. Clicking on the + New button available in the top-right of the browser window will initiate the creation of a new release definition:

Figure 22: Creating a new release definition

Figure 22 shows the Create new release definition user interface. Selecting a release template and providing source artifacts and agent queue information will create a release definition in draft mode. The queue agent selected here will become the default queue agent for first environment within the release definition. If the Empty template is chosen, VSTS will create an environment named Environment 1 with no tasks in it. The same release definition is shown in Figure 23:

Figure 23: Empty release definition

There are three levels of configuration for each release definition-Release level, Environment level and Task level configuration.

• Release level configuration: This configuration affects the entire release definition and execution. It also provides the option to save the release definition from any tab.
  • Environments: This tab lists every environment available in the release definition. It also comprises all the tasks that are part of the environment definition. Inpidual task configuration can be done from this tab.
  • Artifacts: This tab helps configure the link to the artifacts. The artifacts can come from a build execution or from a repository. Build artifacts can be chosen from any project in a Visual Studio account and repository artifacts can choose any repository and select project within the VSTS account. The artifacts type can be Jenkins, Git, TFVC, and GitHub apart from VSTS Build. The Artifacts user interface is shown in Figure 24:
    

    Figure 24: Artifacts tab in release definition

  • Variables: This tab helps make the release definition generic and reusable. They help remove hard coding and change the behavior of a release definition during execution. They also help use the same value in multiple places within the same definition. There is a set of predefined variables available. In addition, new variables can be defined by the user. Each variable has a name and value. Variables can also be encrypted using the secret lock button. These are release-level variables. This is shown in Figure 25. There is another set of variables, known as Environment variables, that are defined at the environment level. We will look at them when we talk about environment level configuration in the following section:
    

    Figure 25: Variables tab in release definition

  • Triggers: Release can be initiated manually or as a scheduled activity or as continuous deployment. If Continuous Deployment is chosen as an option, the availability of any new build version or any new commit/check-in of code (if the continuous integration is configured on the linked build) will start the release process by queuing it to the agent queue. The artifacts configuration determines whether to use the build output or new commits in the code repository to initiate the release. The build version must be chosen manually as an artifact if release is manually initiated. This is shown in Figure 26. Trigger configuration is also available at the environment level and we will look into it when we discuss environment-specific configuration in the following section:
    

    Figure 26: Triggers tab in release definition

  • General: This tab allows you to configure the release name. Every release is given a unique number, and Release name format helps provide more useful names for the releases. The General tab is shown in Figure 27:
    

    Figure 27: General tab in release definition

  • Retention: VSTS stores logs and other information for every release execution. This tab allows to set the number of days for which to retain the release logs.
  • History: VSTS maintains the history of changes made to the release definition, which was not possible in earlier versions of VSTS. This provides version control for the release definition itself and allows reversion back to a previous version and comparison between multiple versions of same definition. The History details are shown in Figure 28:
    

    Figure 28: History tab in release definition

    There is further configuration available through the release definition context menu. Among the more important of them is the Security configuration. Security configuration allows you to provide contributor, administrator, and authoring permissions to different team members. The steps to configure security for the release definition are the same as those for the build definition.

    Clicking on the Release button and then on the Create Release item creates a new release, enlists the environments on agent queues, and starts executing them.

• Environment level configuration: This configuration affects an inpidual environment's definition and execution. Each environment has a button with the symbol: .... Clicking on it and then on any one of the menu items - Assign approvers..., Agent queues, Configure variables, or Deployment conditions, will show the user interface for configuring that environment. Environment Context Menu is shown in Figure 29:
  

  Figure 29: Environment contextual menu

  You can delete an environment definition from its contextual menu, as shown in Figure 30:

  

  Figure 30: Deleting an environment definition

  • Approvals: Approvals play a pivotal role in Release Management because they affect multiple environments that are strategic and important to applications and organizations. Approvals can be sought prior to the execution of an environment definition from a user/group, or they can be configured to be approved automatically. Similarly, we can configure both pre-deployment and post-deployment approvals for an environment. This is shown in Figure 31:
    

    Figure 31: Approvals tab in environment definition

  • Queue: This tab can override the default agent queue. The Demands section lists the capabilities a release agent must have for successful execution of the definition. Release will not execute if the demands are not satisfied. The queue configuration is shown in Figure 32:
    

    Figure 32: Queue configuration in environment definition

  • Variables: The concept of variables is the same as that discussed in the release-level configuration section. Environment also provides the same set of predefined variables as provided at the release level. In addition, new variables can be defined by the user. The release-level variables are accessible at the environment level and can also be overridden here by declaring a variable with the same name. Environment level variables are visible only within the environment they are defined in. This is shown in Figure 33:
    

    Figure 33: Variable configuration in environment definition

  • General: This tab allows the configuration of e-mail notifications and you may also provide a name for the owner of the environment. Skip artifacts download ensures that the artifacts are not downloaded to the agent prior to starting the deployment. Deployment timeout in minutes provides control to VSTS to cancel the environment execution if it does not finish within the time provided. This is shown in Figure 34:
    

    Figure 34: General configuration in environment definition

• Deployment conditions: Environment release can be initiated manually. It can be triggered automatically, after the creation of a release from the release definition, as a scheduled activity, or as part of continuous deployment. This is shown in Figure 35:
  

  Figure 35: Deployment configuration in environment definition

• Task-level configuration: Release definitions have same tasks available for configuration as those available to build definitions. We have already covered this during our discussion about build task level configuration.

Setting up a cloud environment

Azure and VSTS accounts are crucial for implementing DevOps processes. In this chapter, we will look at the steps to create both VSTS and Azure accounts, and in Chapter 5, Building a Sample Application, we will set up the development environment for our application.

Visual Studio Team Services

The primary prerequisite for creating an account with VSTS is to have a Microsoft account. Microsoft accounts were previously known as Live accounts. This is a free account that can be set up through https://signup.live.com/ to access Microsoft services such as Skype, OneDrive, Outlook, Hotmail, and more.

Another way to create a VSTS account is to have a Work or School account, which refers to an enterprise and its e-mail accounts.

One account, either a Microsoft account or a Work or School account, is necessary to create a VSTS account. After provisioning these accounts, browsing through https://go.microsoft.com/fwlink/?LinkId=307137&clcid=0x409 will start a wizard to create a VSTS account. It will ask you to log in with your Microsoft or Work account. After login, pick a unique name through which your VSTS account is identified. This is shown in Figure 36. You can also select the type of repository - Git or Team Foundation Version Control. By clicking on the Change details link, select a preferred region and the process types (CMMI, SCRUM, and Agile) to manage the work. This is shown in Figure 37:

Figure 36: Creating a VSTS account

Click on the Continue button and in a few seconds, your account will be created as shown in Figure 37.

Figure 37: Creating a VSTS account