Part 2 – GRID on Cloud – Azure or AWS?

If you are planning to do some testing and pilot with nVIDIA GRID on one of the two major public cloud providers – Microsoft Azure or Amazon Web Services (AWS), which one would you go for? Answer to it really depends on your objective.

Based on my research so far, it appears to me that Microsoft Azure is a clear cut choice to purchase GPU-optimised cloud compute. Few of the factors (merits) which supports the Azure platform compared to AWS are as follows:

  • Latest underlying host hardware for the GRID solution
  • More granular Compute VM options (cores, memory, disk)
  • Multuple GPU options available – Kepler/Maxwell
  • Higher CUDA cores
  • Higher Number of simultaneous H.264 streams Video Encoding
  • Competitive or lesser pricing for both windows and linux.

To summarize the differences in the platform and configuration between the Azure and AWS cloud offerings, the following tables may be handy checklist to take a quicker decision for your first time customers or otherwise who wants to see the real time performance and user experience:

grid_azure_aws1

Azure vs AWS Server Hardware for GRID Comparison

Azure vs AWS GRID VM Configuration Comparison

References

Azure & AWS Host Comparison

Microsoft Azure References

Azure N-series Availability
Azure pricing
nVIDIA Azure solution brief

AWS References

AWS GPU Instances
AWS GPU VM Guest OS
AWS Marketplace
AWS pricing

Advertisements
Posted in AWS, Azure, GRID | Tagged , , , | 1 Comment

Part 1 – nVIDIA GRID Offerings – quick reference

As you may already be seeing lots of information flowing around in the internet around the GRID virtualization solution delivered by nVIDIA, there have been some quick and significant changes (improvements) over the last few years around the GRID virtualization offerings from nVIDIA GRID integration with VMware, Citrix and Microsoft for the Blast Extreme/PCoIP, HDX/3D Pro and RemoteFX protocol respectively, to deliver Graphics accelerated virtual apps and desktops. Per my opinion, out of the three leading vendors, VMware and Citrix have been aggressive in their commitment compared to Microsoft. However it may be aggressive with the introduction of Windows Server 2016 architecture and beyond.

Having said that, I thought why not just put together a quick reference of the three primary GRID Card offerings from nVIDIA, their use-cases, vGPU profiles, licensing and other requirements. This may come handy for the Sales and the Pre-Sales team to basically engage with their customer base to identify the needs of User Experience at its highest quality apart from the graphics intensive apps & desktop virtualization.

Important Note (Disclaimer): Please be aware while reading this that many of the features, functionalities may not have been tested, validated and thus may change from real-time results. Therefore, I strongly recommend you to leverage (as applicable) the official documentation, whitepaper and blogs from nVIDIA, Citrix, VMware and Microsoft or other reliable sources, as there are continuous changes and updates are being released constantly. Few of the important official documentation are listed in the references section at the bottom of this blog.

  GRID Solution Matrix with VMware, Citrix and Microsoft

GRID Software Editions
 Features & Components GRID Virtual Application (vApp) GRID Virtual PC (vPC) GRID Virtual WorkStation (vWS)
nVIDIA GPU cards supported Tesla M10 (User

Density-Optimized)

Tesla M60 (Performance-Optimized) Tesla M6 (Blade-

Optimized)

Remoting App & Desktop solutions supported XenApp, Horizon Hosted Apps (RDSH/ThinApp) XenDesktop, Horizon View/vSGA, RemoteFX, vDGA Workstation PCs with GPU passthrough, High-end VDI (desktop OS)
Use-cases or user profiles Knowledge/task worker Business/Power User/Medium 3D worker High-end & powerful Designers / 3D app users
Client OS supported N/A Windows Windows/Linux
Maximum Displays N/A 4 4
Maximum Resolution N/A 2560×1600 4096×2160 (4K)
CUDA & OpenCL Supported Yes (only on 8GB 1:1 profiles i.e. M10-8A and M10-8Q) Yes (only on 8GB 1:1 profiles i.e. M60-8A and M60-8Q) Yes (only on 8GB 1:1 profiles i.e. M6-8A and M6-8Q)
DirectX 12, Direct2D, and DirectX Video Acceleration (DXVA) Yes Yes Yes
OpenGL 4.5 Yes Yes Yes
NVIDIA GRID SDK (remote graphics acceleration) Yes Yes Yes
GPU Pass-through Supported Yes (only on 8GB 1:1 profiles) Yes (only on 8GB 1:1 profiles)
Bare-Metal Supported Yes (Only NVIDIA Tesla M6 Hardware supported as primary display device) Yes (Only NVIDIA Tesla M6 Hardware supported as primary display device)
GRID Card Technical Specifications
GPU Card Tesla M10 Tesla M60 Tesla M6
OEM hardware consideration designed for rack and tower servers, optimized for maximum user density per host designed for rack and tower servers, optimized for performance designed for blade servers and converged
Number of GPUs Quad Mid-Level Maxwell Dual High-End Maxwell Single High-End Maxwell
Total NVIDIA CUDA®Cores 2,560 (640 per GPU) 4,096 (2,048 per GPU) 1,536
Total Memory Size 32 GB GDDR5 (8 GB per GPU) 16 GB GDDR5 (8 GB per GPU) 8 GB GDDR5
Max vGPU Instances 64 32 16
Max Power 225 W 300 W 100 W
Form Factor PCIe 3.0 Dual Slot (rack) PCIe 3.0 Dual Slot (rack) MXM (blade)
Board Dimensions 10.5″ x 4.4″ 10.5″ x 4.4″ 3.2″ x 4.1″
Cooling Solution Passive Passive / Active Bare Board
GRID Licensing Model – Concurrent User (CCU)
Licence Type – Option 1 Perpetual (one-time) + SUMS* (first year mandatory)
Licence Type – Option 2 Annual Subscription – pay as you go (yearly renewal)
License Entitlement vApps vPC + vApps (mixed environment) vWS + vApps (mixed environment)
*Support, Updates, and Maintenance Subscriptions (SUMS) ensures that you have 24×7 access to technical support, along with timely software patches, updates, and upgrades. SUMS is included in your NVIDIA GRID software subscription, but is a required one-year add-on if you choose a perpetual license. NVIDIA GRID K1 and K2 GPUs do not require a license to run vGPU.
License Bundle Inclusions
GPU Card Tesla M10 Tesla M60 Tesla M6
Graphics drivers – Windows
Graphics drivers – Windows/Linux Baremetal, Pass-through and vGPU drivers
GRID vGPU host software
VMware vSGA driver
Tesla iromflsh
gpumodeswitch tool for
graphics/compute mode
change
NVIDIA License Manager
1 GRID Virtual App
Edition for RDSH App
hosting
NVIDIA GRID Certified Servers
GPU Card Tesla M10 Tesla M60 Tesla M6
NVIDIA Certified Server Compatibility URL Tesla M10 Tesla M60 Tesla M6
Manufacturer ASRock Rack
ASUS
Dell
Gigabyte
Inspur
Leadtek
Nutanix
Supermicro
Tyan
Advantech
ASRock Rack
ASUS
Cisco
Cubix
Dell
Fujitsu
Gigabyte
Hitachi
HP
Huawei
Inspur
Inventec
Leadtek
Lenovo
Magma
NEC
Nutanix
QCT
Sugon
Supermicro
Themix
Tyan
VDI-Appliance
Amulet Hotkey
Cisco
HP

nVIDIA GRID Profiles & Configuration Matrix

GRID GPU Profile and Configuration Matrix

GRID GPU Profile and Configuration Matrix

API, Open-source programming Language Support Matrix for GRID with VMware, Citrix and Microsoft

GRID – API Support Matrix

Important Notes from above table:
*vSGA is supported only by Horizon
*GPU pass-through requires special hypervisor enablement which is available in VMware vSphere Hypervisor (ESXi) and Citrix XenServer.
*Windows Server 2016 supports NVIDIA GPU pass-through with Discrete Device Assignment (DDA).

 References

nVIDIA GRID licensing, Packaging and User Guides

VMware GPU References

(Note: The reference architecture refers to K1/K2 cards, therefore considering the latest updates and changes, refer to the latest release notes of Horizon View 7.x and nVIDIA GRID M6/10/60 guides)

VMware Terminologies for Virtual GPU solutions

  • NVIDIA GRID vGPU (shared GPU hardware acceleration) Available with vSphere 6.0 and later, this feature allows a physical GPU on an ESXi host to be shared among virtual machines. This feature offers flexible hardware-accelerated 3D profiles ranging from lightweight 3D task workers to high-end workstation graphics power users.
  • Virtual Dedicated Graphics Acceleration (vDGA) Available with vSphere 5.5 and later, this feature dedicates a single physical GPU on an ESXi host to a single virtual machine. Use this feature if you require high-end, hardware-accelerated workstation graphics.
  • Virtual Shared Graphics Acceleration (vSGA) Available with vSphere 5.1 and later, this feature allows multiple virtual machines to share the physical GPUs on ESXi hosts. This feature is suitable for mid-range 3D design, modeling, and multimedia applications.
  • Soft 3D Software-accelerated graphics, available with vSphere 5.0 and later, allows you to run DirectX 9 and OpenGL 2.1 applications without requiring a physical GPU. Use this feature for less demanding 3D applications such as Windows Aero themes, Microsoft Office 2010, and Google Earth.

VMware VMotion Capability: Because NVIDIA GRID vGPU and vDGA use PCI pass-through on the ESXi host, live VMotion is not supported. vSGA and Soft 3D support live VMotion.

Microsoft GPU References

Note: Based on my search online, there are not proper official documentation by Microsoft on RemoteFX, GRID or otherwise. And the blogs are quite old and relies mostly on results gathered from various customer environment and user community.

Citrix GPU References

Notes from Citrix docs:

Support for NVIDIA Kepler architecture. HDX 3D Pro supports NVIDIA GRID cards (see NVIDIA GRID) for GPU pass-through and GPU sharing.

NVIDIA GRID vGPU enables multiple VMs to have simultaneous, direct access to a single physical GPU, using the same NVIDIA graphics drivers that are deployed on non-virtualized operating systems.

Support for VMware vSphere and VMware ESX using Virtual Direct Graphics Acceleration (vDGA) – You can use HDX 3D Pro with vDGA for both RDS and VDI workloads.

Posted in Enterprise Virtualization | Tagged , , , , | 2 Comments

Playing with nVIDIA GRID on VMware and Citrix accelerated graphics remoting solutions – Get Started User Guides

{Firstly, apologies to all the readers for the delay in sharing this through my blog though it is available online. Thought before it disappears online or the URLs goes missing or bad, i should record it in my blog.}

I worked on a GRID project and derived the following attached artifacts  during my tenure at Citrix. I believe these documents will be quite handy and useful for someone who is getting there hands dirty for the first time with nVIDIA GRID proof-of-concept, lab setup or for that matter production roll-out.

Important Note: Since these documents were produced more than couple of years back, lot of the product specific terminologies, version, build etc. might have changed. Therefore, do your due diligence to refer to the official online information for relevance and up to date information.

Part 1: XenServer GPU Pass-through

Part 2: vSphere GPU Pass-through

Part 3: XenServer GPU Virtualization (vGPU)

Part 4: vSphere software GPU (vSGA)

These are also publicly available free to download at the following URLs in case you cannot download from my blog site:

Publications – Whitepaper

Reviewer’s Guide for Remote 3D Graphics Apps:

Part 1: XenServer GPU Pass-through

Part 2: vSphere GPU Pass-through

Part 3: XenServer GPU Virtualization (vGPU)

Part 4: vSphere software GPU (vSGA)

Posted in Enterprise Virtualization, Horizon, VMware View, vSphere, XenApp, XenServer | Tagged , , , | Leave a comment

Securing A+ with NetScaler VPX 11.0 64.34.nc

All traffic will pass through the NetScaler appliance which is secured using HTTPS on 443! Yeah that’s correct! Oh wait, what’s the security team’s audit report stating – my netscaler is not secured it has several gaps found.

Hmm yeah thats correct, I have come across several deployments in my experience where consultants completely miss to secure the netscaler gateway hosting various services such as Exchange/CAS, ADFS SSON, reverse proxy for several web apps, content switching …..list goes on which are vulnerable to various attacks! Enough said,

With this blog, I share my experience with you on how to efficiently secure NetScaler to score A+ on the security report radar!

Couple of reasons for writing the blogs are:

  • NetScaler VPX has some limitations around ciphers and hence scoring A+ is a bit tricky
  • Different build/version of NetScaler requires different ways to obtain higher grades

Note:

As per Citrix documentation, http://docs.citrix.com/en-us/netscaler/11/traffic-management/ssl/customize-ssl-config/config-ecdhe-ciphers.html – The following table lists the ciphers supported on VPX instances and MPX appliances.

However note that for NetScaler 11.0 64.34.nc build, cipher AES-GCM\SHA2 is NOT supported and hence needs to be removed as part of the A+ steps as mentioned at a later step in this blog:

Cipher Suite

VPX

MPX

TLS1-ECDHE-RSA-RC4-SHA

YES

YES

TLS1-ECDHE-RSA-DES-CBC3-SHA

YES

YES

TLS1-ECDHE-RSA-AES128-SHA

YES

YES

TLS1-ECDHE-RSA-AES256-SHA

YES

YES

TLS1.2-ECDHE-RSA-AES128-GCM-SHA256

NO

YES

TLS1.2-ECDHE-RSA-AES256-GCM-SHA384

NO

YES

TLS1.2-ECDHE-RSA-AES-128-SHA256

NO

YES

TLS1.2-ECDHE-RSA-AES-256-SHA384

NO

YES

Steps taken on the NetScaler VPX 11.0 64.34.nc:

  • Perfect Forward Secrecy (PFC) has been configured

Perfect Forward Secrecy protect a session from being decrypted when server key became compromised. For more details, https://en.wikipedia.org/wiki/Forward_secrecy

Configure Diffie-Hellman key (Perfect Forward Secrecy):

Navigate to Traffic Management > Load Balancing > SSL.

Go To Tools > Create Diffie-Hellman (DH) key.

ns1

DH Path: /nsconfig/ssl/dhkey2048.key

  • Size: 2048
  • DH Generator: 2

ns2

Configure DH:

Navigate to NetScaler Gateway Virtual Servers.

Edit your vServer and go to SSL Parameters. Check Enable DH Param.

Browse to the previously created DH Key.

ns3

Alternatively, you can configure  Diffie-Hellman (DH) key from shell:

set ssl vserver <VSERVERNAME> -dh ENABLED -dhFile "/nsconfig/ssl/dhkey2048.key"
  • HTTP Strict Transport Security (HSTS) with long duration deployed Strict Transport Security

STS or HSTS prevents a website for being accessed on another protocol than HTTPS. For more info: http://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security

Create Rewrite Action:

ns4

Create Rewrite Policy:

ns5

Bind Rewrite Policy to NetScaler Gateway virtual server:

ns6

  • Unsupported cipher keys such as AES-GCM\SHA2 have been removed from this VPX 11.0 64.34.nc build

ns7

  • Rivest Cipher 4 (RC4) stream cipher has been removed from the cipher suites as it is no longer recommended and have multiple vulnerabilities found recently.

In 2015, there were speculation that some state cryptologic agencies may possess the capability to break RC4 when used in the TLS protocol. IETF has published RFC 7465 to prohibit the use of RC4 in TLS; Mozilla and Microsoft have issued similar recommendations.

  • Cipher Suites with lower bits such as 112 have been removed from the virtual server configuration.
  • The order of the Cipher Suites has been rearranged to ensure effectiveness and priorities of each cipher keys

ns8

  • Lastly ensure TLS1.1/1.0, SSL versions are disabled in your gateway virtual server

ns10

Rescan (with clear cache) your netscaler gateway URL in qualys ssl labs and voila!

ns9

Also, I would highlight the following folks who have already shared their tips through their blogs:

Scoring an A+ at SSLlabs.com with Citrix NetScaler (the sequel)

Scoring an A+ on SSLLABS.COM with NetScaler 11 VPX

Posted in Enterprise Virtualization | Leave a comment

NetScaler Bug :: invalid admin bind credentials

Scenario

During one of my deployment at a customer site, bumped across this bug around the latest build of NetScaler. Build Version: 11.0 64.34.nc

Under Authentication > Dashboard, the Status will show the error instead of Up/green (assuming that the Firewall rules are in-place/OK):

nsbug

The issue is seen when you try to add the authentication server/profile, in this case it is LDAP, in the add auth page – when you enter the LDAP bind credentials (with special characters) and perform ‘retrieve attributes’ task, the page refreshes shows up as down invalid admin bind credentials.

nsbug1

 

Workaround

Use an account with no special characters (I understand this may not be ideal from a security perspective therefore I tried adding the authentication server from CLI and the outcome still is same. Hopefully this should be addressed in the next release of netscaler build)

Reference forum discussion http://discussions.citrix.com/topic/373583-down-%E2%80%93-invalid-admin-bind-credentials/

Posted in Enterprise Virtualization | Tagged | Leave a comment

Netscaler Access Gateway v11 fails to connect to storefront due to TLSv1.2 schannel errors

Scenario

On a recent customer deployment, I came across this issue where externally, using access gateway connecting to the citrix environment failing, it passed the LDAP authentication stage and then redirection to the backend storefront (LB/server) was not happening, the page simply goes blank (white) with an hour glass.

Environment

  • Netscaler MPX 5500
  • Netscaler version 11.0 build 62.10.nCore
  • Storefront 2.6
  • XenDesktop 7.6
  • Configured to use single FQDN for both internally and externally

Findings

On further investigation, found several errors in the event log in the storefront server:

Log Name: System
Source: Schannel
Date: 11/09/2015 8:04:52 AM
Event ID: 36874
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: <computer name>
Description:
An TLS 1.2 connection request was received from a remote client application, but none of the cipher suites supported by the client application are supported by the server. The SSL connection request has failed.

Searched online and found the following two pointers:

Referred the citrix forum as well – http://discussions.citrix.com/topic/368520-netscaler-11-and-storefront-30-load-balancing-broken/

NS 11.0.62.10.nc release notes suggest that https://www.citrix.com/content/dam/citrix/en_us/documents/downloads/netscaler-adc/NS_11_0_62_10.html

Even though TLS protocol versions 1.1 and 1.2 are not supported by firmware version 1.1, the protocols incorrectly appear as enabled by default on an SSL virtual server.
Workaround: Disable TLS1.1/1.2 explicitly on the virtual server.
[# 576274]

Workaround tried however did not work, then called Citrix support and they suggested to perform the following:

Workaround 1

In Netscaler Access Gateway section, under Sessions > Session Profiles > Edit: change the storefront address in the session profiles from https:// to http:// addresses and re-enable the TLS v1.2/1.1

Workaround 2

After trying the option, externally, the page now redirects to the citrix storefront page and enumerates the apps/desktops however the published desktop launches and throws an error status code 1030. Went ahead and disabled the TLS v1.2/1.1 on the access gateway page and all worked OK!

This might be a concern for security centric organisations and may be there is a better and convincing way to handle this scenario.

Conclusion

Lastly, as of at the time of writing this blog, this is a known issue in NS MPX + v11.0 firmware – TLS/schannel errors in storefront and external access fails to connect to backend storefront LB/server. Citrix indicates Mircosoft do not support TLS v1.2 on the windows storefront server and/or some issues with the NS v11.0 (still unknown, hope this get fixed in near future)

Posted in Enterprise Virtualization, NetScaler Gateway | Tagged , | 2 Comments

XenApp 6.5 – cloud-hosted WebM/MP4 videos may not work in Internet Explorer

Problem Description

In one of my customer site, they are using cloud-hosted videos for their department. Basically, a cloud based video delivery system. The videos are in MP4 and WebM format. There appears to be a problem viewing these videos using Internet Explorer (any version),from within the Citrix environment, however they play as required when using Google Chrome and Firefox. The videos play all OK without any issues outside of the Citrix environment (XenApp 6.5) using IE or any browser.

This indicates that the problem lies within the citrix environment. Somehow due to some policies or something unknown is blocking it to load & play the videos.

Error Message

Error Loading media: File could not be played.

ctx_mp4_1

Cause

This error message is generic and be due to several issues such as HDX flash redirection, H.264 codec conversion failure etc. But in this scenario it was due to the following:

Debugging

To reproduce the scenario, follow below steps on any XenApp server directly (RDP’ing):

  • Open the website http://<cloud-hostedvideowebsite>/
  • Go to IE Settings > F12 Developer Tools > Emulation tab
  • Change Document Mode from Edge(default) to 8

ctx_mp4_2

The page will refresh (reset) automatically with IE 8 engines (don’t close the F12 panel)

Just click on any videos and it should play

Resolution & Workaround

Resolution (for webpage owner/developer)

Configure the webpage to restrict a it to a document mode (8 in this case) supported by an older version of Windows Internet Explorer. You need to consider the x-ua-compatible header, which allows a webpage to be displayed as if it were viewed by an earlier version of the browser. Please follow this MSDN library page titled Specifying legacy document modes for further steps on how to do it. May be it will be good if there can be a separate web-page hosting for Citrix environment, as the site works all OK in non-citrix environment.

ctx_mp4_3

Extract from one of the MS page: If you are using the X-UA-Compatible META tag you want to place it as close to the top of the page’s HEAD as possible. Internet Explorer begins interpreting markup using the latest version. When Internet Explorer encounters the X-UA-Compatible META tag it starts over using the designated version’s engine. This is a performance hit because the browser must stop and restart analyzing the content.

The best practice is an X-UA-Compatible HTTP Header. Adding the directive to the response header tells Internet Explorer what engine to use before parsing content begins. This must be configured in the web site’s server. Custom HTTP Headers can be added in Internet Information Server through the management console.

You can also add custom HTTP headers in the ASP.NET web.config:

<configuration>
<system.webServer>
<httpProtocol>
<customHeaders>
<add name=”X-UA-Compatible” value=”IE=edge” />
</customHeaders>
</httpProtocol>
</system.webServer>
</configuration>

HTTP Headers may also be added to the web application’s response by the application’s code. In ASP.NET custom headers can be added to the response using the AddHeader method. The following shows how to programmatically add the X-UA-Compatible header.

HttpContext.Response.AddHeader(“X-UA-Compatible”, “IE=edge”);

Workaround (for System Admins)

Open the browser in Enterprise Mode instead of the Standard Mode (default). To enable this, you need to configure a group policy (GPO) in Active Directory or in your computer’s Local Policy (if no AD infrastructure in-place). Follow the below steps to configure and enable the policy:

Turn on Enterprise Mode and use a site list

Open your Group Policy editor and go to the Administrative Templates\Windows Components\Internet Explorer\Use the Enterprise Mode IE website list setting.

ctx_mp4_4

To get started with this policy, you need to create a site list using the Enterprise Mode Site List Manager tool which can be downloaded from the Microsoft download page here.

Add sites to the Enterprise Mode site list using the Enterprise Mode Site List Manager. To add a site to your compatibility list,

In the Enterprise Mode Site List Manager tool, click Add.

Type the URL for the website that’s experiencing compatibility problems, like <domain>.com or <domain>.com/<path> into the URL box. You don’t need to include the http:// or https:// designation. The tool will automatically try both versions during validation.

ctx_mp4_5

Pick Enterprise Mode if the site should use the new, modified browser configuration or pick Default IE if it should use the latest version of Internet Explorer.

ctx_mp4_6

Click Save to validate your website and to add it to the site list for your enterprise. If your site passes validation, it’s added to the global compatibility list. If the site doesn’t pass validation, you’ll get an error message explaining the problem. You’ll then be able to either cancel the site or ignore the validation problem and add it to your list anyway.

On the File menu, go to where you want to save the file, and then click Save to XML.

ctx_mp4_7

You can save the file locally or to a network share. However, you must make sure you deploy it to the location specified in your registry key.

This is how the XML file looks like:

ctx_mp4_8

Add the URL of the XML file to the GPO: Administrative Templates\Windows Components\Internet Explorer\Use the Enterprise Mode IE website list

ctx_mp4_9

You may save the XML file on a webserver (HTTP), network share or locally on any computer/server. As examples:

HTTP location: “SiteList”=”http://localhost:8080/sites.xml”

Local network: “SiteList”=”\\network\shares\sites.xml”

Local file: “SiteList”=”file:///c:\\Users\\<user>\\Documents\\testList.xml”

Once the GPO is applied, ensure you go to each of the XenApp servers and run ‘gpupdate’ to ensure the policy is propagated to the servers.

And in the registry on the XenApp server, you should see the following entry:

ctx_mp4_13

To verify the webpage opens in Enterprise mode, launch the IE as a published app and open the video-hosting webpage in question. Go to Tools > F12 Developer Tools, under Emulation tab, you should see the Document mode is set to 5(Default)

ctx_mp4_12

Posted in Enterprise Virtualization, XenApp | Tagged | Leave a comment