VM Scale-out Code Examples

There is a new blog about automating VM scale-out migration recommendations on the corporate Tintri site. Along with that script I wrote another script to set VM affinity for VM scale-out migration rules. These rules allow you to exclude VMs from being considered for migration.

Let’s look at a use case. Suppose there is one Flash VMstore, and 2 hybrid VMstores in a VMstore pool. VDI is running in your Flash VMstore, and you don’t what the VDI VMs to migrate to the hybrid VMstores, therefore, the VDI VMs are put into a Service Group by name pattern. Unfortunately, the script would have to be executed periodically because affinity is set at the VM level.  When a VM is added to the Service Group, the script needs to be executed.

VMs are excluded with --affinity set to ‘never‘.  VMs are specified by a service group, --sg, by a list of VM name, --vms, or by pattern matching with --name. To clear the affinity set --affinity to ‘clear‘.

So let’s look at the script, set_reco_vm_affinity.py.

# Let's get to work
    if (len(vms) > 0):
        print("Collecting VMs from list")
        vm_uuids += get_vms_in_list(server_name, session_id, vms)

    if (service_group != ""):
        print("Collecting VMs from service group")
        sg_uuid = get_sg_by_name(server_name, session_id, service_group)
        if (sg_uuid == ""):
            raise tintri.TintriRequestsException("Can't find service group " + service_group)

        vm_uuids += get_vms_by_sg(server_name, session_id, sg_uuid)

    if (vm_contains_name != ""):
        print("Collecting VMs from name")
        vm_uuids += get_vms_by_name(server_name, session_id, vm_contains_name)

    if (len(vm_uuids) == 0):
        raise tintri.TintriRequestsException("No VMs to set rules")

    if debug_mode:
        count = 1
        for uuid in vm_uuids:
            print(str(count) + ": " + uuid)
            count += 1

    # Process according to affinity
    if (affinity == "never"):
        set_vm_affinity_never(server_name, session_id, vm_uuids)
    elif (affinity == "clear"):
        clear_vm_affinity(server_name, session_id, vm_uuids)
        raise tintri.TintriRequestsException("Bad affinity rule: " + affinity)

except tintri.TintriRequestsException as tre:
except tintri.TintriApiException as tae:

In the above snippet, the code figures out the list of VMs and processes for the specified affinity.

The crux of this script is how to set the affinity:

# A helper function  that sets the VM affinity rule for migration recommendations.
def set_vm_affinity(server_name, session_id, vm_uuids, affinity_rule):
    url = "/v310/vm/"

    for vm_uuid in vm_uuids:
        rule_url = url + vm_uuid + "/affinity"

        r = tintri.api_put(server_name, rule_url, affinity_rule, session_id)
        if r.status_code != 204:
            tintri.api_logout(server_name, session_id)
            message = "The HTTP response for put affinity rule to the server is not 204."
            raise tintri.TintriApiException(message, r.status_code,
                                            rule_url, str(affinity_rule), r.text)

# Set the VM affinity rule to never for a list of VMs.
def set_vm_affinity_never(server_name, session_id, vm_uuids):
    print("Setting " + str(len(vm_uuids)) + " VMs to never migrate")

    affinity_rule = \
        {"typeId" : beans + "vm.VirtualMachineAffinityRule",
         "ruleType" : "NEVER"

    set_vm_affinity(server_name, session_id, vm_uuids, affinity_rule)

# Clear the VM affinity rule for a list of VMs
def clear_vm_affinity(server_name, session_id, vm_uuids):
    print("Clearing " + str(len(vm_uuids)) + " VMs affinity rules")

    affinity_rule = \
        {"typeId" : beans + "vm.VirtualMachineAffinityRule",

    set_vm_affinity(server_name, session_id, vm_uuids, affinity_rule)

In the function set_vm_affinity(), we see the API invoke /v310/vm/{vm_uuid}/affinity. For each VM UUID in the input list, the API is invoked. There are two helper functions set_vm_affinity() and clear_vm_affinity to facilitate.

Although it has been discussed before, you can review how to get the VMs in a service group with this snippet of code.

# Return a list of VM UUIDs based on a service group name.
def get_vms_by_sg(server_name, session_id, sg_uuid):
    vm_uuids = []
# Get a list of VMs, but return a page size at a time
vm_filter = {"includeFields" : ["uuid", "vmware"],
             "serviceGroupIds" : sg_uuid
vm_uuids = get_vms(server_name, session_id, vm_filter)

return vm_uuids

# Return VM items constrained by a filter.
def get_vms(server_name, session_id, vm_filter):
vm_uuids = []

# Get a list of VMs, but return a page size at a time
get_vm_url = "/v310/vm"
vm_paginated_result = {'next' : "offset=0&limit=" + str(page_size)}

# While there are more VMs, go get them, and build a dictionary
# of name to UUID.
while 'next' in vm_paginated_result:
    url = get_vm_url + "?" + vm_paginated_result['next']

    r = tintri.api_get_query(server_name, url, vm_filter, session_id)
    print_debug("The JSON response of the VM get invoke to the server " +
                server_name + " is: " + r.text)

    # For each VM in the page, print the VM name and UUID.
    vm_paginated_result = r.json()
    print_debug("VMs:\n" + format_json(vm_paginated_result))

    # Get the VMs
    items = vm_paginated_result["items"]
    for vm in items:

return vm_uuids

Before I close this post, I would like to mention that you can post API questions on Tintri’s Hub. Until next time,

– Rick –

Tintri On-line Resources


I wanted to point to two Tintri resources that are available to you on-line.  First, is Tintri Resources, which contain videos, white papers, webinars, analyst reports, and data sheets


You can refine your selection by selecting a topic.  Current topics for selection are:

  • Backup and Recovery
  • Flash
  • Hyper-V
  • Private Cloud
  • Quality of Service (QoS)
  • Server Virtualization
  • Software Development
  • Tintri Global Center (TGC)
  • Tintri VMstore
  • VMstack
  • VVol.

For example, in checking out the Backup and Recovery topic, there are white papers on Best Practices with VMware vCenter, Symantec NetBackup, CommVault Simpana, and Veeam.

Another on-line resource is Tintricity Hub discussions. There are public and private forums. You need to sign-up to the Hub to access the private forums. In the private forums,  you can ask questions, and discuss Tintri REST APIs, and Tintri Automation Toolkit.  There is a general discussion group for everything else. Currently, the only public forum is on Data Protection.

The Tintricity Hub also has Challenges, Referrals, and Rewards.  By answering challenges and reading content, you collect points fro rewards. Rewards include gift cards, charity donations, one hour of free consulting, and Tintri SWAG.

When you have some time, please check-out our Tintri resources,

– Rick –


New Python API Examples


There are 3 updates to the Python API examples on GitHub. One is about updating data IP addresses on a VMstore.  The other 2 examples are on VMstore appliance information collection.

Updating  Data IP Addresses

I wrote an Python example, set_data_ip.py, that displays and/or updates the data IP addresses for a VMstore.  The script always displays the current IP addresses.  Depending on the options, one data IP address is added or removed. The main crux of the code is similar to the code setting DNS. This is described in  more detail in this week’s “Ask Rick” blog.

Collecting VMStore Information

I had previously added an example, appliance_info.py, which displays the following:

| Info        | Value                    |
| Product     | Tintri VMstore           |
| Model       | T445                     |
| All Flash   | False                    |
| OS version  | |
| API version | v310.51                  |

I have moved this code to appliance_status.py, because it displays short concise VMstore or TGC status.

Now that we have appliance_status.py, I re-created appliance_info.py to display the following information:

  • VMstore status
  • Appliance components
  • Failed components if any
  • IP addresses: admin, data, and replication
  • Controller status
  • Disk information

This script imports the prettytable module.

Until next time,

– Rick –

Adding VMs to a Service Group


A colleague, Adam  Cavaliere, wrote a Python script that adds VM name from a CSV file to a TGC Service Group. The script, set_service_group_members.py is now on Tintri’s GitHub site.

This script asks for the user name and password interactively instead being on the command line.  The command line only has the TGC server name, the TGC service group to put the names into, and the CSV file name that contains the VM names.

Let’s look at the salient part of the code.

if service_group_exists:
    target_sg_api = "/v310/servicegroup/" + target_sg + "/members/static"
    print("Service Group Found")

payload = {'typeId':'com.tintri.api.rest.v310.dto.CollectionChangeRequest', \
           'objectIdsAdded': uuid_list

# Set the static members.
tt.api_put(server_name, target_sg_api, payload, session_id)

Here we have the API invoke to update the static members of the specified service group with a list of VM UUIDs.  The list of VM UUIDs are added to payload before the api_put() call.

The code below obtains the target_sg for the update service group static members API URL  It performs a GET servicegroup and looks for the specified service group and extracts the service group UUID.

# Get List of Service Groups
api = "/v310/servicegroup"
results = tt.api_get(server_name, api, session_id)
service_groups = results.json()

service_group_exists = False

# setup API for adding VMs to proper Service Group
for item in service_groups["items"]:
    if item["name"] == service_group:
        service_group_exists = True
        target_sg = item["uuid"]["uuid"]

From a list VM names, the VM UUIDs are collected in the get_vm_uuids() function. This function takes as input a list of VM names and returns a list of found VM names and their UUIDs.  The GET vm API is is used.  I have discussed this in previous blogs. The UUID list is used in the payload above.

That pretty much covers it,

– Rick –

Obtain a VM CSV Report


A new Python API example, get_vm_report.py, has been uploaded to our Tintri GitHub site.  This new script outputs a CSV report file on all the VMs known to the Tintri server.  (A Tintri server is currently a VMstore or TGC.)  The script also takes a CSV input file that controls which columns are returned.

The tintri-api-examples/reports directory includes vm_report_fields.csv, vm_latency_report_fields.csv, and of course get_vm_report.py.  Let’s discuss the CSV files first.

The file vm_report_fields.csv is a template that contains all the fields or columns that can be included in the VM report.  Each line contains the field name and a short description. To customize your report, copy vm_report_fields.csv to a new file and uncomment the field lines in the new file to be included in the report.  An example of this is vm_latency_report_fields.csv.  Upon examination, you’ll see that the following field lines are uncommented: VmName, TotalLatency, NewtorkLatency, StorageLatency, and DiskLatency.

The Python script reads the input CSV for the fields, and places the fields in the DTO. The API is invoked and a URL is returned.  Using the URL, the report is downloaded.  The tintri_1_1.py library was modified to include a file download function.

Until next time,

– Rick –



Videos, Videos, and more Videos

Now that our big Tintri Launch is done, I would like to point to some videos that you might find interesting and educational.  These videos review the Tintri launch as well as provide reference materials for earlier products. I would recommend watching these videos in full screen mode.

First off, the Storage Field Day at Tintri is now online which includes 6 videos. This set of videos present a comprehensive view of our new Tintri products.

  1. Tintri Product Overview and Vision with Kieran Harty
  2. Tintri VM-Aware Storage and Quality of Service Demo with Justin Lauer
  3. Tintri VM Scale Out Demo with Mark Gitter
  4. Tintri Analytics Demo with Brandon Salmon
  5. Tintri SyncVM demo with Christopher Slater
  6. Tintri Wrap-up Discussion

If Brandon Salmon’s demo on Tintri Analytics left you wanting more, then check-out Andrew Large‘s Tintri Analytics Video Series. The series contains an introduction, creating application definitions, predicting resource utilization, and planing future VM deployments.

If you want a starter video on VM-scale out, try Sunil Muralidhar‘s, video on Tintri VM Scale-Out: Optimal VM distribution.This video is definitely an introduction.

Next up, Sunil Muralidhar, demonstrates the features of TGC Service Groups.  TGC Service Groups allow a collection of VMs across multiple VMstores to have the same protection and QoS policies.  There is a Python API TGC Service Group QoS example as well as a PowerShell example on GitHub.

In a set of 6 videos, Rob Girard covers cloning and Sync VM manually and with automation.

  1. Manually cloning VMs
  2. Automating cloning VMs with scripts
  3. Reconfiguring VMs for testing
  4. Manually refreshing VMs with production data
  5. Automatically refreshing VMs with production data
  6. Automatically refreshing VMs from a single snapshot

Finally, Christopher Slater is up again in a another set of 6 videos that discusses databases and Tintri.

  1. Performance visualization and isolation
  2. Snapshots in seconds
  3. Clone a database in a few clicks
  4. Restore a database with SyncVM
  5. VM level replication made easy
  6. Set QoS policy for an individual VM

If you want more, you can go to Tintri’s video resource page, and Tintri’s YouTube Channel. I realize that this is a lot to go through, but you can keep this blog handy and view the videos at your leisure;

Happy viewing,

– Rick –

Tintri’s Triple Play

Welcome to Tintri’s May 2016 launch.

I’m just going to quickly cover three launch subjects:

  • Increased all-flash VMstores
  • VM scale out
  • Predictive Analysis

Now the T5000 all-flash arrays have increased storage capacity, approximately a 4 times increase over the old T5000 storage capacity.

T5000 New All-flash Maximum Capacity
Model Max Raw Capacity Max Effective Capacity
T5040 23 TB 77 TB
T5060 46 TB 154 TB
T5080 92 TB 308 TB

Effective capacity is a result of dedupe, compression, and zero block optimization. See the T5000 All-Flash data sheet for more information.

Along with increased storage capacity, VMstores can be federated together into a pool of storage for VM scale-out. Both hybrid and all-flash can be in the same storage pool for resource allocation and resource management. Via the Tintri Global Center (TGC), deep workload analysis identifies VMs that are growing rapidly or heavily using IOPS and makes VM storage migration recommendations.  See Introducing Tintri VM Scale-out for more details.

Even with VM scale-out, you’ll want capacity and performance predictions for planning and forecasting.  With Tintri Analytics, you’ll be able crunch numbers for up to 160,000 virtual machines, get application insights, and throw away your crystal ball.  For more details, see the Tintri Analytics data sheet.

And if you’re hungry for more, you can register for a webinar on May 19 from Kieran Harty, one of Tintri’s co-founders.  Sign up here.

Until next time,

– Rick –

Setting VMstore Maintenance Mode


It has been a while since I’ve written a blog, but here is one that sets the VMstore maintenance mode. Hopefully there will be more coming at a faster pace.

One of Tintri’s customers wanted to set the VMstore’s maintenance mode. When the VMstore maintenance mode is enabled, alert e-mails are not sent except to support@tintri.com.  Theses support@tintri.com e-mails are marked with “maintenance mode”.

Setting maintenance mode allows you to change network configuration or upgrade VMstore without the VMstore sending alert e-mails.  That way the administrator will not be bothered with extraneous e-mails.  Tintri support still receives e-mails just in case something really important occurs, like a disk failure.

I wrote Python and PowerShell examples. Both are interactive, that is, they ask the user if the maintenance mode should be flipped before flipping the enable flag. The maintenance mode flag is located in the maintenance mode data transfer object (DTO) in the appliance DTO.
ApplianceMaintenanceMode DTO:

{ "typeId": "com.tintri.api.rest.v310.dto.domain.beans.hardware.ApplianceMaintenaneMode", "endTime": DateTime, "isEnabled": <Boolean>, "startTime": DateTime, }

Appliance DTO:

{ "typeId": "com.tintri.api.rest.v310.dto.domain.Appliance",
  "errorString": <String>,
  "fanDescription": <String>,
  "lastUpdatedTime": DateTime,
  "localAlias": <String>,
  "localPassword": <String>,
  "powerDescription": <String>,
  "redundancyDescription": <String>,
  "temperatureDescription": <String>,
  "alertCounts": ApplianceAlertCounts,
  "allowSnapshotIncompleteVm": ApplianceAllowSnapshotIncompleteVm,
  "components": [ApplianceComponent],
  "configIps": [ApplianceIp],
  "controllers": [ApplianceController],
  "customizationInfo": ApplianceCustomizationInfo,
  "dateTimeConfig": ApplianceDateTime,
  "disks": [ApplianceDisk],
  "diskEncryptionInfo": ApplianceDiskEncryptionInfo,
  "dnsConfig": ApplianceDns,
  "emailConfig": ApplianceEmail,
  "info": ApplianceInfo,
  "ipmiConfig": ApplianceIpmi,
  "lacpConfig": ApplianceLacp,
  "maintenanceMode": ApplianceMaintenanceMode,
  "operationalInfo": ApplianceOperationalInfo,
  "rbacExternalConfig": RbacExternalConfig,
  "snmpConfig": ApplianceSnmp,
  "supportConfig": ApplianceSupport,
  "syslogForwarding": ApplianceSyslogForwarding,
  "temperatures": [Temperature],
  "upgradeInfo": ApplianceUpgradeInfo,
  "uuid": Uuid,

First let’s look at the Python example. The code uses the Setting DNS Primary at Scale as an example, because both perform a PUT on the /v310/appliance resource.  When setting the maintenance mode, start and end times are required.  The VMstore GUI sets a 6 hour duration and the code follows this precedent.

 now = datetime.datetime.now()
 add_6 = now + datetime.timedelta(hours=6)
 time_zone = my_timezone()
 now_str = now.isoformat() + time_zone
 add_6_str = add_6.isoformat() + time_zone

When resetting the maintenance mode the start and end times should not be present, so the code reflects this.

if (new_is_enabled):
    # Create the maintenance mode DTO for enabling.
    new_maint_mode_info = \
      {"typeId": "com.tintri.api.rest.v310.dto.domain.beans.hardware.ApplianceMaintenanceMode",
       "endTime" : add_6_str,
       "isEnabled" : new_is_enabled,
       "startTime" : now_str
    # Create the maintenance mode DTO for disabling.
    new_maint_mode_info = \
      {"typeId": "com.tintri.api.rest.v310.dto.domain.beans.hardware.ApplianceMaintenanceMode",
       "isEnabled" : new_is_enabled,

The request DTO is used to send to the Appliance DTO with the new maintenance mode DTO filled with values to set.

# Create the Appliance object wit the new ApplianceDns DTO.
new_appliance = \
  {"typeId": "com.tintri.api.rest.v310.dto.domain.Appliance",
   "maintenanceMode": new_maint_mode_info

# Create the Request object with the Appliance DTO.
request = \
  {"typeId": "com.tintri.api.rest.v310.dto.Request",
   "objectsWithNewValues": [new_appliance],
   "propertiesToBeUpdated": ["maintenanceMode"]

# Invoke the appliance API to set the maintenance mode.
r = tintri.api_put(server_name, url, request, session_id)

The rest of the code gets and prints the maintenance mode. The code is locate on our GitHub site at set_maintenance_mode.py.

The PowerShell example follows the Python example closely.  This is due to bug 42915 in the PowerShell Toolkit and the fact that I was unable to use the Tintri Session ID from the toolkit in Invoke-RestMethod().  For me this example was a lesson in using Invoke-RestMethod().  I used Mircosoft’s library and trial and error.

One of things I did was create some helper functions, like tintriVersion(), tintriLogin(), and tintriLogout().  Here is tintriLogin():

Set-Variable JSON_CONTENT "application/json; charset=utf-8"
Set-Variable APPLIANCE_URL "/api/v310/appliance/default"

function tintriLogin
$loginUri = "https://$($server)/api/v310/session/login"
$loginDict = @{typeId="com.tintri.api.rest.vcommon.dto.rbac.RestApiCredentials";
$loginBody = $loginDict | ConvertTo-Json 
$resp = Invoke-RestMethod -sessionVariable session -Method Post -Uri $loginUri -Body $loginBody -ContentType $JSON_CONTENT
return $session

The key here is to obtain the session variable which is done with -sessionVariable session. See the code snippet above in the login invoke. Note, the sessionVariable parameter value, session, doesn't have a dollar sign. Later Invoke-RestMethod() calls use the session variable with the WebSession parameter. An example is below.

$url = "https://$($server)$($APPLIANCE_URL)/maintenanceMode"
$resp = Invoke-RestMethod -Uri $url -Method Get -WebSession $session -ContentType $JSON_CONTENT

Again, we get the time and a six hour delta:

$now = Get-Date
$add6 = $now.AddHours(6)
$nowStr = $now.ToString("yyyy-MM-ddTHH:mmzzz")
$add6Str = $add6.ToString("yyyy-MM-ddTHH:mmzzz")

And we build the ApplianceMaintenanceMode DTO depending on whether the maintenance mode is enabled or not.

if ($newIsEnabled) {
    # Create the maintenance mode object for enabling.
    $newMaintModeInfo = @{typeId = "com.tintri.api.rest.v310.dto.domain.beans.hardware.ApplianceMaintenanceMode";
                          endTime = $add6Str;
                          isEnabled = $newIsEnabled;
                          startTime = $nowStr
else {
    # Create the maintenance mode object for disabling.
    $newMaintModeInfo = @{typeId = "com.tintri.api.rest.v310.dto.domain.beans.hardware.ApplianceMaintenanceMode"; 
                          isEnabled = $newIsEnabled

Finally, the Request DTO is created, and the /v310/appliance resource is invoked.

GitHub API Python Examples Updated


The API python examples on the Tintri GitHub site have been updated.

The library, python_1_1.py has been updated to disable SSL warnings. To facilitate this, a new method, api_version(), has been added to the library. It replaces the /info GET invoke.

All the API python client examples have been updated to use the python_1_1.py library and api_version() method. Since api_version() returns the same values as /info, the change is minimal. Here is an example diff:

 - r = tintri.api_get(server_name, '/info')
 + r = tintri.api_version(server_name)
      json_info = r.json()
      product_name = json_info['productName']

A new API client example, set_qos_tgc_service_groups.py is now available. This example shows how to use the new QoS TGC Service Group APIs. With the new APIs, configuring QoS for Service Groups has been simplified. The blog, TGC Service Groups and QoS Redux discusses this new QoS example.


– Rick –

PowerShell 2.5 Toolkit Released


It has been a while since I have posted. While I’ve been on vacation and catching-up, my colleague was working on a new Tintri Automation Toolkit for PowerShell 2.5. Today it went up on Tintri’s support site for your downloading pleasure. If you haven’t visited the support site in a while, you’ll notice that it has changed. Once on the support download page, find “Tintri Automation Toolkit 2.5” and click on it. An information frame will appear on the right. Click on “Download MSI” to download the new toolkit.

The major features are:
• Upgrade VMstore(s) given the RPM file
• Configure appliance settings (Data IPs, DNS, Autosupport, Alerts)
• Configure Hypervisor Managers on a VMstore
• Set Service Group QoS
• Support for SyncVM File-Level-Restore
• Manage VMstores on a TGC
• Set quota on SMB shares
• Support for PowerCLI 6

A more information is on my Tintri blog. For me, the top 3 features are: Tintri command search, setting QoS in TGC Service Groups, and updating VMstores from a downloaded RPM file.

Let me know if you have questions, or have coded new scripts you want posted on our GitHub site.


– Rick –