Create your own Environment in Microsoft Fabric

Managing and maintaining hardware and software settings is difficult in the ever-evolving world of data science and big data. This is where Microsoft Fabric steps in, introducing the concept of “Environment” to simplify and streamline these processes. In this blog post, we will create, configure, and utilize the environment in Microsoft Fabric. This will provide a step-by-step guide to set up and use this environment within their organization.

What is the Environment in Microsoft Fabric?

In Microsoft Fabric, an environment is a comprehensive container for your hardware and software settings within Spark. It provides a unified interface where you can select the desired Spark runtime, install libraries, and configure Spark compute settings and properties. This simplifies the process of managing, maintaining, and sharing these configurations.

Why Environment was introduced in Microsoft Fabric?

The concept of “Environment” was introduced in Microsoft Fabric to simplify and streamline the process of managing, maintaining, and sharing both hardware and software settings within Spark.

Here are some key reasons why environments were introduced:

1. Unified Interface: The environment serves as a comprehensive container for your hardware and software settings within Spark. This unified interface allows you to select the desired Spark runtime, install libraries, and configure Spark compute settings and properties.

2. Simplified Configuration Management: Managing libraries and Spark compute configurations have been integral features of the environment. With these core functionalities, you can tailor distinct configuration sets.

3. Resource Management: The Resources folder facilitates the management of small resources during the development phase. Uploaded files within the environment are accessible from various notebooks once they are attached to the same environment.

4. Collaboration: Environment sharing is now available, allowing you to collaborate seamlessly. When you share an environment item, recipients automatically receive read permission.

5. Version Control: Microsoft Fabric environments offer Git integration for seamless source control with Azure DevOps. This allows you to track changes, revert to previous versions, and maintain a history of your configurations.
   

Create a New Environment in Microsoft Fabric

There are two different ways to create an environment in Microsoft Fabric:

1. Standard: You can create a new environment through the new section of your Microsoft Fabric workspace

2. During selection: You can select the environment for your notebook, Spark job definition, and the workspace default, there will be an option to create a new environment.

Here are the steps to create a new environment:

STEP 1: Access the Microsoft Fabric Portal

Navigate to the Microsoft Fabric portal. On the left panel, click on the “Workspace” option.

STEP 2: Creation of a New Environment

In the Microsoft Fabric Workspace section, locate and click the “+ New” button. A drop-down menu will appear. Select “Environment” from this list.

STEP 3: Name Your Environment

A pop-up box will appear. Type your desired name in the provided field and click “Create” to proceed.

STEP 4: Configure Your Environment

After naming your environment, you’ll be presented with three main options to configure your environment:

Spark Runtime: You select the desired Spark runtime for your environment. Each runtime comes with its default settings and preinstalled packages.

Libraries: Libraries are packages or modules you can use in your Spark applications. In Microsoft Fabric, you can manage two types of libraries:

• Public Libraries: These are libraries available in public repositories. You can search for and install these libraries directly from the Microsoft Fabric interface.

• Custom Libraries: These are libraries that you or your organization have created. You can upload these libraries from your local directory to the Microsoft Fabric environment. Microsoft Fabric supports custom library files in .whl, .jar, and .tar.gz formats.

Spark Compute: This includes two main components:

• Compute: This refers to the compute resources that your Spark job applications will use. You can choose from various pools created by workspace admins and capacity admins.
  

• Spark Properties: These configurations control the behavior of Spark job applications. You can manage these properties to be effective in Spark sessions.

Resources: This is a shared storage that can change in real time.

STEP 5: Save and Publish Your Environment

The Spark compute and libraries configurations are required for the publishing to be effective, while resources are shared storage that can change in real-time. Click on the “Publish” button to finalize your environment.

Important:

• If you are updating the runtime of an environment with existing configurations or libraries, you must republish the contents based on the updated runtime version.  

• The publishing fails if the existing configurations or libraries are incompatible with the newly updated runtime version.

• You must remove the incompatible configurations or libraries and publish the environment again.

Conclusion

In conclusion, the concept of “Environment” in Microsoft Fabric is a powerful tool that simplifies and streamlines the process of managing, maintaining, and sharing both hardware and software settings within Spark. It provides a unified interface for selecting the desired Spark runtime, installing libraries, and configuring Spark compute settings and properties.

Furthermore, Microsoft Fabric environments offer Git integration for seamless source control with Azure DevOps, promoting a true single data source for analysis and decision-making. Whether you’re a data scientist, a data engineer, or a developer, Microsoft Fabric environments can enhance your productivity and foster collaboration in your data projects.