Changing QoS during Veeam Backups

Tintri’s QoS is a great tool to limit storage bandwidth per VM, but what happens when a backup occurs?  Well, QoS will also limit your backup storage IOPS. If this is an issue in your environment, you might consider clearing the QoS during backup.

As an example of clearing QoS when a backup occurs,  a PowerShell script is now available that interacts with Veeam Backup and Replication.  The script, Veeam_Backup_Tintri_QoS.ps1 is available on GitHub.  The file, README.md has the information needed to run the script.

Basically the script polls the Veeam backup server every 10 seconds. When a Veeam backup job has started, the script finds the associated storage.  If the storage is on a VMstore and QoS is set, then the QoS is cleared.  After the job is done, the script restores the QoS settings.  A TGC server is required since the script connects only to a TGC.

Below is a diagram of Veeam_Backup_Tintri_QoS.ps1 (Backup Script) running in the Veeam backup server.

.veeam_tgc_diagram

However the backup script could be in a separate VM.

This script could possibly be used as a template for other backup vendors.  Results may very.  Veeam made it straight-forward with a PowerShell plug-in.

Until next time,

– Rick –

tintri_veeam_logos_small

 

PowerShell VM CSV Report

Greetings,

A customer via a colleague, Satinder Sharma, wanted to have a CSV output from Get-TintriVM. Their attempt was:

Get-TintriVM  | Export-Csv WS_inventory.csv

This does nothing.  After some experimentation and assistance from Dhruv Vermula, we have this:

Get-TintriVM | Select-Object {$_.vmware.name}, {$_.Uuid.UuId} | Export-csv vm2.csv

This works, but the columns are “$_.vmware.name” and “$_Uuid.Uuid”, which is not very friendly.  So I created created a small script, VmCsvReport, that is similar to a blog post “Obtain a VM CSV Report“.  This script creates nice columns; for example, “VM Name” instead of “$_.vmware.name” and creates a CSV file.

The script uses @{Expression} to create alias columns.

 $ex = @{Expression={$_.vmware.name};label="VM Name"},
       @{Expression={$_.stat.sortedstats.LatencyTotalMs};label="Total Latency"},
       @{Expression={$_.stat.sortedstats.LatencyNetworkMs};label="Network Latency"},
       @{Expression={$_.stat.sortedstats.LatencyStorageMs};label="Storage Latency"},
       @{Expression={$_.stat.sortedstats.LatencyDiskMs};label="Disk Latency"}

The output of Get-TintriVM is piped into Select-Object using the aliases in $ex.  Then $results is piped to the Export-Csv cmdlet to create a CSV file.  This could be done in one line, but I broke it up so that I could easily examine the output in debug.

$result = Get-TintriVM -TintriServer $conn | Select-Object $ex
$result | Export-Csv $csv_file

The specified VM information is in the CSV file which Excel can read quite easily.

Unfortunately, the columns are hard-coded in the script, because the PowerShell Toolkit does not have a cmdlet that maps the CSV download API.  I couldn’t suss-out a way to have configurable columns similar “Obtain a VM CSV Report“. If you know a way, please let me know.

Until next time,

– Rick –

Automation, Cloud, Containers, and PySDK

Greetings,

November has started with a bang of announcements and publications:

  1. Tintri Raises the Bar for Enterprise Cloud Build on Web Services Architecture and RESTful APIs” discusses Tintri’s architecture, automation, container support, analytics, and public cloud integration.  There will be a webinar on November 10th at 11 am Pacific with Kieran Harty and Steve Herrod.
  2. The Register’s article on Tintri’s Chatbot , automation, cloud, and container support talks about Tintri’s Web Services approach that makes everything API-accessible which makes automation , container support, public cloud integration, and chat-bot possible.
  3. ComputerWeekly.com published an article of “Tintri adding VMware and Flocker persistent container storage” which discusses Tintri’s container Flocker support in more detail.
  4. Tintri announced the release of the Python SDK (PySDK). This PySDK was used in the above mentioned chat-bot. With PySDK, Tintri will be able to integrate easier with automation platforms like OpenStack.
  5. Tintri published a white paper on Tintri’s vRealize Orchestrator Plugin. Now VMware’s vRealize orchestrator can manage Tintri storage.  This allows scripts in vRealize to invoke Tintri management APIs.

This is just the beginning of Tintri’s cloud announcements.  More will be coming in the next months.  Until next time,

– Rick –

Using Tintri APIs to Build Tintri Anywhere

Greetings,

We have a customer in Japan, Adways, the used Tintri APIs, to build a Slack-bot than communicates with a Tintri VMstore.  A Slack client sends a command message to the Slack server just like messages are sent between people.  The Slack server sends a message to the Tintri bot that invokes the appropriate Tintri API.  See the picture below:

adways_slackbot_api

This allows the VMstore to be managed by a cellphone. Here is an example:

@tintribot:tintri tintri-001 show appliance_info

The result output is:


+-------------+--------------------------+
| Info        | Value                    |
+-------------+--------------------------+
| Product     | Tintri VMstore           |
| Model       | T540                     |
| OS version  | 3.2.2.1-6163.33224.16186 |
| API version | v310.21                  |
+-------------+--------------------------+

If this interests you, please check out Masaaki Hatori’s blog. I had to use Google Chrome to translate it from Japanese to English.

Regards,

– Rick –

VM Scale-out Code Examples

There is a new blog about automating VM scale-out migration recommendations on the corporate Tintri site. Along with that script I wrote another script to set VM affinity for VM scale-out migration rules. These rules allow you to exclude VMs from being considered for migration.

Let’s look at a use case. Suppose there is one Flash VMstore, and 2 hybrid VMstores in a VMstore pool. VDI is running in your Flash VMstore, and you don’t what the VDI VMs to migrate to the hybrid VMstores, therefore, the VDI VMs are put into a Service Group by name pattern. Unfortunately, the script would have to be executed periodically because affinity is set at the VM level.  When a VM is added to the Service Group, the script needs to be executed.

VMs are excluded with --affinity set to ‘never‘.  VMs are specified by a service group, --sg, by a list of VM name, --vms, or by pattern matching with --name. To clear the affinity set --affinity to ‘clear‘.

So let’s look at the script, set_reco_vm_affinity.py.

# Let's get to work
try:
    if (len(vms) > 0):
        print("Collecting VMs from list")
        vm_uuids += get_vms_in_list(server_name, session_id, vms)

    if (service_group != ""):
        print("Collecting VMs from service group")
        sg_uuid = get_sg_by_name(server_name, session_id, service_group)
        if (sg_uuid == ""):
            raise tintri.TintriRequestsException("Can't find service group " + service_group)

        vm_uuids += get_vms_by_sg(server_name, session_id, sg_uuid)

    if (vm_contains_name != ""):
        print("Collecting VMs from name")
        vm_uuids += get_vms_by_name(server_name, session_id, vm_contains_name)

    if (len(vm_uuids) == 0):
        raise tintri.TintriRequestsException("No VMs to set rules")

    if debug_mode:
        count = 1
        for uuid in vm_uuids:
            print(str(count) + ": " + uuid)
            count += 1

    # Process according to affinity
    if (affinity == "never"):
        set_vm_affinity_never(server_name, session_id, vm_uuids)
    elif (affinity == "clear"):
        clear_vm_affinity(server_name, session_id, vm_uuids)
    else:
        raise tintri.TintriRequestsException("Bad affinity rule: " + affinity)

except tintri.TintriRequestsException as tre:
    print_error(tre.__str__())
    sys.exit(4)
except tintri.TintriApiException as tae:
    print_error(tae.__str__())
    sys.exit(5)

In the above snippet, the code figures out the list of VMs and processes for the specified affinity.

The crux of this script is how to set the affinity:

# A helper function  that sets the VM affinity rule for migration recommendations.
def set_vm_affinity(server_name, session_id, vm_uuids, affinity_rule):
    url = "/v310/vm/"

    for vm_uuid in vm_uuids:
        rule_url = url + vm_uuid + "/affinity"

        r = tintri.api_put(server_name, rule_url, affinity_rule, session_id)
        if r.status_code != 204:
            tintri.api_logout(server_name, session_id)
            message = "The HTTP response for put affinity rule to the server is not 204."
            raise tintri.TintriApiException(message, r.status_code,
                                            rule_url, str(affinity_rule), r.text)
        sys.stdout.write(".")
    print("")

# Set the VM affinity rule to never for a list of VMs.
def set_vm_affinity_never(server_name, session_id, vm_uuids):
    print("Setting " + str(len(vm_uuids)) + " VMs to never migrate")

    affinity_rule = \
        {"typeId" : beans + "vm.VirtualMachineAffinityRule",
         "ruleType" : "NEVER"
        }

    set_vm_affinity(server_name, session_id, vm_uuids, affinity_rule)

# Clear the VM affinity rule for a list of VMs
def clear_vm_affinity(server_name, session_id, vm_uuids):
    print("Clearing " + str(len(vm_uuids)) + " VMs affinity rules")

    affinity_rule = \
        {"typeId" : beans + "vm.VirtualMachineAffinityRule",
        }

    set_vm_affinity(server_name, session_id, vm_uuids, affinity_rule)

In the function set_vm_affinity(), we see the API invoke /v310/vm/{vm_uuid}/affinity. For each VM UUID in the input list, the API is invoked. There are two helper functions set_vm_affinity() and clear_vm_affinity to facilitate.

Although it has been discussed before, you can review how to get the VMs in a service group with this snippet of code.

# Return a list of VM UUIDs based on a service group name.
def get_vms_by_sg(server_name, session_id, sg_uuid):
    vm_uuids = []
# Get a list of VMs, but return a page size at a time
vm_filter = {"includeFields" : ["uuid", "vmware"],
             "serviceGroupIds" : sg_uuid
            }
vm_uuids = get_vms(server_name, session_id, vm_filter)

return vm_uuids

# Return VM items constrained by a filter.
def get_vms(server_name, session_id, vm_filter):
vm_uuids = []

# Get a list of VMs, but return a page size at a time
get_vm_url = "/v310/vm"
vm_paginated_result = {'next' : "offset=0&limit=" + str(page_size)}

# While there are more VMs, go get them, and build a dictionary
# of name to UUID.
while 'next' in vm_paginated_result:
    url = get_vm_url + "?" + vm_paginated_result['next']

    r = tintri.api_get_query(server_name, url, vm_filter, session_id)
    print_debug("The JSON response of the VM get invoke to the server " +
                server_name + " is: " + r.text)

    # For each VM in the page, print the VM name and UUID.
    vm_paginated_result = r.json()
    print_debug("VMs:\n" + format_json(vm_paginated_result))

    # Get the VMs
    items = vm_paginated_result["items"]
    for vm in items:
        vm_uuids.append(vm["uuid"]["uuid"])

return vm_uuids

Before I close this post, I would like to mention that you can post API questions on Tintri’s Hub. Until next time,

– Rick –

Tintri On-line Resources

Greetings,

I wanted to point to two Tintri resources that are available to you on-line.  First, is Tintri Resources, which contain videos, white papers, webinars, analyst reports, and data sheets

Tintri_Resources.

You can refine your selection by selecting a topic.  Current topics for selection are:

  • Backup and Recovery
  • Flash
  • Hyper-V
  • Private Cloud
  • Quality of Service (QoS)
  • Server Virtualization
  • Software Development
  • Tintri Global Center (TGC)
  • Tintri VMstore
  • VMstack
  • VVol.

For example, in checking out the Backup and Recovery topic, there are white papers on Best Practices with VMware vCenter, Symantec NetBackup, CommVault Simpana, and Veeam.

Another on-line resource is Tintricity Hub discussions. There are public and private forums. You need to sign-up to the Hub to access the private forums. In the private forums,  you can ask questions, and discuss Tintri REST APIs, and Tintri Automation Toolkit.  There is a general discussion group for everything else. Currently, the only public forum is on Data Protection.

The Tintricity Hub also has Challenges, Referrals, and Rewards.  By answering challenges and reading content, you collect points fro rewards. Rewards include gift cards, charity donations, one hour of free consulting, and Tintri SWAG.

When you have some time, please check-out our Tintri resources,

– Rick –

 

New Python API Examples

Greetings,

There are 3 updates to the Python API examples on GitHub. One is about updating data IP addresses on a VMstore.  The other 2 examples are on VMstore appliance information collection.

Updating  Data IP Addresses

I wrote an Python example, set_data_ip.py, that displays and/or updates the data IP addresses for a VMstore.  The script always displays the current IP addresses.  Depending on the options, one data IP address is added or removed. The main crux of the code is similar to the code setting DNS. This is described in  more detail in this week’s “Ask Rick” blog.

Collecting VMStore Information

I had previously added an example, appliance_info.py, which displays the following:

+-------------+--------------------------+
| Info        | Value                    |
+-------------+--------------------------+
| Product     | Tintri VMstore           |
| Model       | T445                     |
| All Flash   | False                    |
| OS version  | 4.2.0.1-7524.41058.18773 |
| API version | v310.51                  |
+-------------+--------------------------+

I have moved this code to appliance_status.py, because it displays short concise VMstore or TGC status.

Now that we have appliance_status.py, I re-created appliance_info.py to display the following information:

  • VMstore status
  • Appliance components
  • Failed components if any
  • IP addresses: admin, data, and replication
  • Controller status
  • Disk information

This script imports the prettytable module.

Until next time,

– Rick –

Adding VMs to a Service Group

Greetings,

A colleague, Adam  Cavaliere, wrote a Python script that adds VM name from a CSV file to a TGC Service Group. The script, set_service_group_members.py is now on Tintri’s GitHub site.

This script asks for the user name and password interactively instead being on the command line.  The command line only has the TGC server name, the TGC service group to put the names into, and the CSV file name that contains the VM names.

Let’s look at the salient part of the code.

if service_group_exists:
    target_sg_api = "/v310/servicegroup/" + target_sg + "/members/static"
    print("Service Group Found")

payload = {'typeId':'com.tintri.api.rest.v310.dto.CollectionChangeRequest', \
           'objectIdsAdded': uuid_list
}

# Set the static members.
tt.api_put(server_name, target_sg_api, payload, session_id)

Here we have the API invoke to update the static members of the specified service group with a list of VM UUIDs.  The list of VM UUIDs are added to payload before the api_put() call.

The code below obtains the target_sg for the update service group static members API URL  It performs a GET servicegroup and looks for the specified service group and extracts the service group UUID.

# Get List of Service Groups
api = "/v310/servicegroup"
results = tt.api_get(server_name, api, session_id)
service_groups = results.json()

service_group_exists = False

# setup API for adding VMs to proper Service Group
for item in service_groups["items"]:
    if item["name"] == service_group:
        service_group_exists = True
        target_sg = item["uuid"]["uuid"]
        break

From a list VM names, the VM UUIDs are collected in the get_vm_uuids() function. This function takes as input a list of VM names and returns a list of found VM names and their UUIDs.  The GET vm API is is used.  I have discussed this in previous blogs. The UUID list is used in the payload above.

That pretty much covers it,

– Rick –

Obtain a VM CSV Report

Greetings,

A new Python API example, get_vm_report.py, has been uploaded to our Tintri GitHub site.  This new script outputs a CSV report file on all the VMs known to the Tintri server.  (A Tintri server is currently a VMstore or TGC.)  The script also takes a CSV input file that controls which columns are returned.

The tintri-api-examples/reports directory includes vm_report_fields.csv, vm_latency_report_fields.csv, and of course get_vm_report.py.  Let’s discuss the CSV files first.

The file vm_report_fields.csv is a template that contains all the fields or columns that can be included in the VM report.  Each line contains the field name and a short description. To customize your report, copy vm_report_fields.csv to a new file and uncomment the field lines in the new file to be included in the report.  An example of this is vm_latency_report_fields.csv.  Upon examination, you’ll see that the following field lines are uncommented: VmName, TotalLatency, NewtorkLatency, StorageLatency, and DiskLatency.

The Python script reads the input CSV for the fields, and places the fields in the DTO. The API is invoked and a URL is returned.  Using the URL, the report is downloaded.  The tintri_1_1.py library was modified to include a file download function.

Until next time,

– Rick –

 

 

Videos, Videos, and more Videos

Now that our big Tintri Launch is done, I would like to point to some videos that you might find interesting and educational.  These videos review the Tintri launch as well as provide reference materials for earlier products. I would recommend watching these videos in full screen mode.

First off, the Storage Field Day at Tintri is now online which includes 6 videos. This set of videos present a comprehensive view of our new Tintri products.

  1. Tintri Product Overview and Vision with Kieran Harty
  2. Tintri VM-Aware Storage and Quality of Service Demo with Justin Lauer
  3. Tintri VM Scale Out Demo with Mark Gitter
  4. Tintri Analytics Demo with Brandon Salmon
  5. Tintri SyncVM demo with Christopher Slater
  6. Tintri Wrap-up Discussion

If Brandon Salmon’s demo on Tintri Analytics left you wanting more, then check-out Andrew Large‘s Tintri Analytics Video Series. The series contains an introduction, creating application definitions, predicting resource utilization, and planing future VM deployments.

If you want a starter video on VM-scale out, try Sunil Muralidhar‘s, video on Tintri VM Scale-Out: Optimal VM distribution.This video is definitely an introduction.

Next up, Sunil Muralidhar, demonstrates the features of TGC Service Groups.  TGC Service Groups allow a collection of VMs across multiple VMstores to have the same protection and QoS policies.  There is a Python API TGC Service Group QoS example as well as a PowerShell example on GitHub.

In a set of 6 videos, Rob Girard covers cloning and Sync VM manually and with automation.

  1. Manually cloning VMs
  2. Automating cloning VMs with scripts
  3. Reconfiguring VMs for testing
  4. Manually refreshing VMs with production data
  5. Automatically refreshing VMs with production data
  6. Automatically refreshing VMs from a single snapshot

Finally, Christopher Slater is up again in a another set of 6 videos that discusses databases and Tintri.

  1. Performance visualization and isolation
  2. Snapshots in seconds
  3. Clone a database in a few clicks
  4. Restore a database with SyncVM
  5. VM level replication made easy
  6. Set QoS policy for an individual VM

If you want more, you can go to Tintri’s video resource page, and Tintri’s YouTube Channel. I realize that this is a lot to go through, but you can keep this blog handy and view the videos at your leisure;

Happy viewing,

– Rick –